Test compile big openSCAD model

classic Classic list List threaded Threaded
29 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Test compile big openSCAD model

JBoehm
This post has NOT been accepted by the mailing list yet.
Hello everybody,

during these weeks I am developing a program for 3d-printing algebraic surfaces. Up to now, I can triangulate them using CGAL and now want to process the triangulated surfaces with openSCAD. Concretely I want to replace every triangle in the triangulation by a triangular prism of height 0.2 and form the union over all these prisms to get a 3d-printable surface with non-vanishing thickness. The openSCAD-File to do this is prepared by my program, now i have "only" to compile it with openSCAD. The file contains ~37k triangular prisms, described as polyhedron(..) and I can not render it on any of my computers due to memory limitations (I have only a 2GB RAM machine for openSCAD at hand). Before buying a new machine (or trying it on Amazon EC2) I thought it might be a good idea to ask here in the forum, if someone with a big machine could give my file a test-run with openSCAD, to find out, how much memory is really needed for this task. The file is available from my homepage

http://www.aviduratas.de/model.scad

Maybe one could save memory by rearranging the building process in the file (spliting it into several smaller blocks and building a union over them later), but my knowledge of openSCAD currently is too small to make experiments like that useful.

Season's Greetings and many thanks in advance

Jürgen Böhm

Reply | Threaded
Open this post in threaded view
|

Re: Test compile big openSCAD model

jpmendes

Hi,

I have 4GB and could only process the first 20000 lines.
I changed the caches sizes and render elements in OpenSCAD preferences advanced menu by 4x.
Those modifications didn't produce any changes in the result. It aborts always at line 20017.
Task manager process appear to consume only about 1/10th of the memory available.



jpmendes
Reply | Threaded
Open this post in threaded view
|

Re: Test compile big openSCAD model

MichaelAtOz
Administrator
In reply to this post by JBoehm
As jp said the full list produces an error, seems to be some fixed parser limit:

> ERROR: Parser error in line 20017: memory exhausted
> ERROR: Compilation failed!

I first split it in 1/2, that compiled, but rendering ran out of memory.

I then wrapped every 100 (of all 37K) in a union(), that cleared the parser memory limit & compiled:


mmm...chips...

Rendering now, will take a while to find out if it hits my memory limit.
Admin - email* me if you need anything,
or if I've done something stupid...
* click on my MichaelAtOz label, there is a link to email me.

Unless specifically shown otherwise above, my contribution is in the Public Domain; to the extent possible under law, I have waived all copyright and related or neighbouring rights to this work.
Obviously inclusion of works of previous authors is not included in the above.


The TPP is no simple “trade agreement.” Fight it! http://www.ourfairdeal.org/ time is running out!
Reply | Threaded
Open this post in threaded view
|

Re: Test compile big openSCAD model

dbvanhorn
In reply to this post by jpmendes

I can try when i get home.  64g available

On Dec 19, 2015 5:52 PM, "jpmendes" <[hidden email]> wrote:

Hi,

I have 4GB and could only process the first 20000 lines.
I changed the caches sizes and render elements in OpenSCAD preferences
advanced menu by 4x.
Those modifications didn't produce any changes in the result. It aborts
always at line 20017.
Task manager process appear to consume only about 1/10th of the memory
available.

<http://forum.openscad.org/file/n15230/Capture1.jpg>

jpmendes



--
View this message in context: http://forum.openscad.org/Test-compile-big-openSCAD-model-tp15226p15230.html
Sent from the OpenSCAD mailing list archive at Nabble.com.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Test compile big openSCAD model

doug.moen
In reply to this post by jpmendes
This is hard to diagnose without source.

It might be a syntax error that is silently gobbling up the rest of the file? Or a bug in Openscad, in which case I have no clue.

On Saturday, 19 December 2015, jpmendes <[hidden email]> wrote:

Hi,

I have 4GB and could only process the first 20000 lines.
I changed the caches sizes and render elements in OpenSCAD preferences
advanced menu by 4x.
Those modifications didn't produce any changes in the result. It aborts
always at line 20017.
Task manager process appear to consume only about 1/10th of the memory
available.

<http://forum.openscad.org/file/n15230/Capture1.jpg>

jpmendes



--
View this message in context: http://forum.openscad.org/Test-compile-big-openSCAD-model-tp15226p15230.html
Sent from the OpenSCAD mailing list archive at Nabble.com.

_______________________________________________
OpenSCAD mailing list
<a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;Discuss@lists.openscad.org&#39;)">Discuss@...
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org



_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
tp3
Reply | Threaded
Open this post in threaded view
|

Re: Test compile big openSCAD model

tp3
On 12/20/2015 02:49 AM, doug moen wrote:
> This is hard to diagnose without source.
>
The source link is in the first post which did not make
it to the mailing list.

@JBoehm: If you already have the triangulated surface, it
would be much much more efficient to just put that into a
single polyhedron.
Trying to union 37000 polyhedrons might be a test what is
faster: CGAL union or the sun burning up all it's mass.

ciao,
  Torsten.


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
-- Torsten
Reply | Threaded
Open this post in threaded view
|

Re: Test compile big openSCAD model

JBoehm
This post has NOT been accepted by the mailing list yet.
@JBoehm: If you already have the triangulated surface, it
would be much much more efficient to just put that into a
single polyhedron.

I do not quite get, what you mean by "put that into a single polyhedron". The
surface S from which I start is a union of triangles in 3d space and I want to compute, what is called
the "offset surface" of this surface S. It consists of the boundary of all points p in R^3 which
have a distance to the surface S, called d(p, S) <= r for a certain r (which is in my case r=0.1).

If one replaces every triangle by its associated triangular prism
with height parallel to the normal of the triangle one gets an approximation of this "offset surface"

To be more accurate, one would need to insert certain wedges for the edges and certain unions of tetrahedra for the vertices of the original surface S, but I omitted this step up to now, because it
is complicated and the effort would be wasted, if it showed up that rendering in openSCAD would
take infeasible long or comsume impossible amounts of memory.


Trying to union 37000 polyhedrons might be a test what is
faster: CGAL union or the sun burning up all it's mass.

This is a good comparison, but now I have some hope to be faster than the sun: I am rerunning
the file with the modifications of making a union() every 100 polyhedra on my 2GB machine and watching the memory consumption, I noticed that virtual memory use is constantly going up, but resident memory stabilized at 1.7 GB and the machine is still responsive. So it seems, the program is constantly working in this 1.7GB area and depositing results in virtual memory. Now if one polyhedron takes 1 second, corresponding to 1 billion machine operations it should be done in 37000s = 10.2 h. If I would trust my argument about memory consumption, I could now go to bed and look again Sunday afternoon if I was to optimistic in assessing one second/polyhedron.

greetings,

Jürgen
Reply | Threaded
Open this post in threaded view
|

Re: Test compile big openSCAD model

thehans
Instead of extruding every triangle along its normal, can you calculate the normal at each point? Moving the points along their normals, then building polyhedrons out of that would at least avoid the "dry lake bed" look.  

But also as Torsten was saying, if you could do it all as a single polyhedron it would be best. The triangles on the surfaces could be calculated as above, moving points along normals, then all you need is to know which vertices make up the "edge" of the entire surface, and make a triangle strip along the edge between the top and bottom surfaces to close out the polyhedron.  Also at that point it might be just as simple to export it as STL directly, its a very basic format and very similar to the polyhedron module.
Reply | Threaded
Open this post in threaded view
|

Re: Test compile big openSCAD model

Parkinbot
thehans wrote
Instead of extruding every triangle along its normal, can you calculate the normal at each point? Moving the points along their normals, then building polyhedrons out of that would at least avoid the "dry lake bed" look.  
This is the point. A surface looking like that, is not the manifold you'd want to render. At least you should provide data where common points of adjacent triangles are identical (your data has numerical errors). Addionally it is advisable to separate upper and lower points and provide a sorted index for upper and lower border point pairs. Knitting this into a top side, a bottom side and a border side from this data is easy.
If this is not possible a viable approach would be to write code that analyses your data and produces a representation like this.

upper_points =[[x1, y1, z1], ... ];
lower_points =[[x1, y1, z1], ... ];
upper_triags = [[n1, n2, n3], ...];
lower_triags = [[n1, n2, n3], ...];
border_points=[[upper1,lower2] ...];   // sorted


Reply | Threaded
Open this post in threaded view
|

Re: Test compile big openSCAD model

dbvanhorn
I also could only parse the first 20017 lines, even with 64G.  It took only a couple seconds.
Reply | Threaded
Open this post in threaded view
|

Re: Test compile big openSCAD model

dbvanhorn
PS: Task manager didn't even see a blip in memory usage, looks like some artificial limit.
 
W10 64 bit OS, I7-3930K  64G
Reply | Threaded
Open this post in threaded view
|

Re: Test compile big openSCAD model

dbvanhorn
Ok, I tried wrapping all the calls to rendertri in a union, and it processes all then crashes.
Still no real memory consumption.

Small surgery in notepad to create two unions for the calls to rendertri, roughly 50/50.
With two unions, I can preview.   I don't grok why unions are working here, I wouldn't expect that, but ok..

What I got in preview was a disc shape, looked like some curves in the disc. I don't know what I should see, so maybe the unions broke something.

Launched render at 5:55 am
Last message: Rendering Polygon Mesh using CGAL
5:55 AM Memory 6% and CPU at 11%..  ???
6:00 AM Memory 7% and CPU at 10%   Where are all the horsepowers going??
Doing some other stuff on the system, all seems normal, and very lightly loaded.
Checking task manager, OpenSCAD is using 1.264 G and a very slow climb over time.
6:05 AM Memory 7% CPU 10% 1.467G
Interesting that this can execute so slowly and not tie up the machine big time.
It acts as if there's some time delay involved, and not the "sit and spin" type, but actually using a hardware or system timer to pace things.
6:10 AM Memory 8% CPU 10% 1.661G
6:15 AM Memory 8% CPU 10% 2.070G


Reply | Threaded
Open this post in threaded view
|

Re: Test compile big openSCAD model

jon_bondy
If you have 8 cores, you are maxing out one of the cores at 12%, and
OpenSCAD cannot use the other cores.  So maybe it is running flat out.

Jon

On 12/20/2015 8:15 AM, dbvanhorn wrote:

> Ok, I tried wrapping all the calls to rendertri in a union, and it processes
> all then crashes.
> Still no real memory consumption.
>
> Small surgery in notepad to create two unions for the calls to rendertri,
> roughly 50/50.
> With two unions, I can preview.   I don't grok why unions are working here,
> I wouldn't expect that, but ok..
>
> What I got in preview was a disc shape, looked like some curves in the disc.
> I don't know what I should see, so maybe the unions broke something.
>
> Launched render at 5:55 am
> Last message: Rendering Polygon Mesh using CGAL
> 5:55 AM Memory 6% and CPU at 11%..  ???
> 6:00 AM Memory 7% and CPU at 10%   Where are all the horsepowers going??
> Doing some other stuff on the system, all seems normal, and very lightly
> loaded.
> Checking task manager, OpenSCAD is using 1.264 G and a very slow climb over
> time.
> 6:05 AM Memory 7% CPU 10% 1.467G
> Interesting that this can execute so slowly and not tie up the machine big
> time.
> It acts as if there's some time delay involved, and not the "sit and spin"
> type, but actually using a hardware or system timer to pace things.
> 6:10 AM Memory 8% CPU 10% 1.661G
> 6:15 AM Memory 8% CPU 10% 2.070G
>
>
>
>
>
>
> --
> View this message in context: http://forum.openscad.org/Test-compile-big-openSCAD-model-tp15226p15243.html
> Sent from the OpenSCAD mailing list archive at Nabble.com.
>
> _______________________________________________
> OpenSCAD mailing list
> [hidden email]
> http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
>
>
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 2016.0.7294 / Virus Database: 4489/11216 - Release Date: 12/20/15
>
>


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Test compile big openSCAD model

JBoehm
This post has NOT been accepted by the mailing list yet.
In reply to this post by thehans
I hope you get to see this message, as I am still getting the yellow marking, saying I am not subscribed to the mailing list, although I have completed the subscription process even twice.

Anyway:
thehans wrote
Instead of extruding every triangle along its normal, can you calculate the normal at each point? Moving the points along their normals, then building polyhedrons out of that would at least avoid the "dry lake bed" look.  
Now I finally got the idea and also understand what Torsten was saying - as I have the normal at every vertex available, I can extrude by this normal for all adjacent faces to this vertex instead of extruding every face by its own normal. I was somewhat blinded for this simple solution, because I always tried to compute the "exact offset surface" by stubbornly replacing every triangle by a triangular prism, every edge by a cylinder and every point by a sphere (and unifying all this with openSCAD). I also searched the internet for papers on offset surface computation and was led into more and more complex algorithms for doing exactly, what in my special case could be - as your proposal showed - done quite easily approximately.

I have programmed your simple solution with dedicated normal to every vertex and it looks of course much better, no "dry lake bed" anymore. I also rendered (F6) a patch of the new file, consisting of the first 2000 polyhedra. It was done in about 15 minutes, but unfortunately the result was flagged as "non-manifold", which I do not quite understand, where this is coming from, as we have a very well behaved union of polyhedra.

Nevertheless, thank you and all the others, who answered my question, very much for taking the time to put me on the right track for this problem.

Greetings,

Jürgen

P.S. I posted the new model.scad again on my homepage

http://www.aviduratas.de/model.scad
Reply | Threaded
Open this post in threaded view
|

Re: Test compile big openSCAD model

dbvanhorn
In reply to this post by dbvanhorn

1:44 still rendering. Memory climbing slowly. Cpu still about 10%.


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Test compile big openSCAD model

MichaelAtOz
Administrator
In reply to this post by MichaelAtOz
MichaelAtOz wrote
Rendering now, will take a while to find out if it hits my memory limit.
That hit my limit of 8GB (4GB+4GB swap).

It (the 100 union'ed version) has been running for 1/2 a day on my bigger box. (8 core 3.5GHz 16BG+swap raid SSD)
Just checked it this AM, unfortunately it went to sleep at some time...
It is about to hit commit of 7GB, at this stage the peak working set is 5.3GB and current WS <2GB.

(OpenSCAD progress meter showing 0/1000)
Admin - email* me if you need anything,
or if I've done something stupid...
* click on my MichaelAtOz label, there is a link to email me.

Unless specifically shown otherwise above, my contribution is in the Public Domain; to the extent possible under law, I have waived all copyright and related or neighbouring rights to this work.
Obviously inclusion of works of previous authors is not included in the above.


The TPP is no simple “trade agreement.” Fight it! http://www.ourfairdeal.org/ time is running out!
Reply | Threaded
Open this post in threaded view
|

Re: Test compile big openSCAD model

dbvanhorn

What is eating so much time? The cpu is practically at idle.   I run em sims that peg the cpu and graphics card for hours to days and make a very effective room heater.   This isnt even twitching the needle.


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Test compile big openSCAD model

MichaelAtOz
Administrator
dbvanhorn wrote
What is eating so much time? The cpu is practically at idle.   I run em
sims that peg the cpu and graphics card for hours to days and make a very
effective room heater.   This isnt even twitching the needle.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
OpenSCAD is single threaded and does not use GPU either.
So best is the highest GHz CPU and plenty of memory.
Admin - email* me if you need anything,
or if I've done something stupid...
* click on my MichaelAtOz label, there is a link to email me.

Unless specifically shown otherwise above, my contribution is in the Public Domain; to the extent possible under law, I have waived all copyright and related or neighbouring rights to this work.
Obviously inclusion of works of previous authors is not included in the above.


The TPP is no simple “trade agreement.” Fight it! http://www.ourfairdeal.org/ time is running out!
Reply | Threaded
Open this post in threaded view
|

Re: Test compile big openSCAD model

dbvanhorn

I know that but its not even fully tasking one processor.


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Test compile big openSCAD model

doug.moen
If the arguments to a CSG operation like union are so large they don't fit in primary cache, then I'd expect cache thrashing, and a big slowdown as the CPU will be waiting on main memory.

On Sunday, 20 December 2015, david vanhorn <[hidden email]> wrote:

I know that but its not even fully tasking one processor.


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
12