for() loop iteration limitation & list comprehensions

classic Classic list List threaded Threaded
28 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

for() loop iteration limitation & list comprehensions

davidconeff
I am attempting to figure out how to do operations with "large" sets (>10,000 elements), which presents a couple problems in OpenSCAD that I believe there is a solution to, but I can't quite figure it out. I posted earlier about issues with for() being limited to an iteration range of 10,000, looking back this seems like a reasonable limitation since it points out that you're probably doing something inefficient that should otherwise be done with list comprehensions or something that is not stack-intensive.

Example code:

function generate_random_polar(n=5001, r=1)=[for (i=[0:1:n-1]) [r, rands(0,360,1)[0]]];
function polar_to_xy(r_theta=[1,0]) = [r_theta[0]*cos(r_theta[1]),r_theta[0]*sin(r_theta[1])];
list1 = concat(generate_random_polar(),generate_random_polar());
list2 = [for (x = list1) polar_to_xy(x)];

Problem 1: Algorithmic generation of a list with greater than 10,000 elements

The only way I've figured out how to do this is simple concatenation of two more or more smaller lists that are generated by a function call with <10,000 iterations in the for() loop. For lists of random points, no big deal. If I was trying to generate a set of ordered points with certain spaces in a certain canonical order, this could get more complicated to have to "batch" the function calls. Is there a better way?

Problem 2: Performing actions that require access to the inside of a list element (e.g., a coordinate pair)

Say I have a list of coordinates (x,y), and that list is greater than 10,000 elements. How would I loop over the entire list and perform an action such as plotting a cube at that point?, e.g.:

cube_size = 0.01;
for(i = [0:1:len(list2)-1]) {translate([list2[i][0],list2[i][1],0]) cube(cube_size);}


List comprehensions in OpenSCAD are, from what I understand, limited to taking a list as input, performing an operation on the individual elements (and even doing the operation may be conditional depending on the contents of the element), and returning a list. "Actions" can't be performed such as translate(); and cube();, but I can perform math on the elements and return a list of modified elements (e.g., convert from polar to cartesian coordinates).

List comprehensions seem to be implemented in such a way that they are not limited to 10,000 element lists (I don't see any errors when I input lists bigger than that in this example code), so it must have been written in the back-end to not create a huge stack I assume. I don't mind switching to this syntax for operations on lists - it is no doubt more efficient - but on the end of a) creating, and b) doing something with lists that large after the math is done, there doesn't seem to be a way to do that, since both would require "action" syntax that I can only get to work in a normal for() loop. Is there a way around this with a better use of OpenSCAD's language features that I just don't understand well enough yet, or am I stuck with batching the for() loop calls to sets with less than 10,000 elements?

Thanks,
David








_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: for() loop iteration limitation & list comprehensions

nophead
The for loop limit is not really anything to do with stack use, that would be a recursion limit. I think it is just an arbitrary limit to catch infinite loops or massive trees of objects that will cause CGAL to explode in time and or memory use. A list on the other hand is a much smaller data structure than a CGAL object.

If you have a practical use for such massive trees of objects then perhaps the limit can be increased. How many do you need and are you going to need to F6 the result? F5 might be OK but F6 with 10000 objects doesn't seem practical.



On 8 March 2017 at 19:47, David Coneff <[hidden email]> wrote:
I am attempting to figure out how to do operations with "large" sets (>10,000 elements), which presents a couple problems in OpenSCAD that I believe there is a solution to, but I can't quite figure it out. I posted earlier about issues with for() being limited to an iteration range of 10,000, looking back this seems like a reasonable limitation since it points out that you're probably doing something inefficient that should otherwise be done with list comprehensions or something that is not stack-intensive.

Example code:

function generate_random_polar(n=5001, r=1)=[for (i=[0:1:n-1]) [r, rands(0,360,1)[0]]];
function polar_to_xy(r_theta=[1,0]) = [r_theta[0]*cos(r_theta[1]),r_theta[0]*sin(r_theta[1])];
list1 = concat(generate_random_polar(),generate_random_polar());
list2 = [for (x = list1) polar_to_xy(x)];

Problem 1: Algorithmic generation of a list with greater than 10,000 elements

The only way I've figured out how to do this is simple concatenation of two more or more smaller lists that are generated by a function call with <10,000 iterations in the for() loop. For lists of random points, no big deal. If I was trying to generate a set of ordered points with certain spaces in a certain canonical order, this could get more complicated to have to "batch" the function calls. Is there a better way?

Problem 2: Performing actions that require access to the inside of a list element (e.g., a coordinate pair)

Say I have a list of coordinates (x,y), and that list is greater than 10,000 elements. How would I loop over the entire list and perform an action such as plotting a cube at that point?, e.g.:

cube_size = 0.01;
for(i = [0:1:len(list2)-1]) {translate([list2[i][0],list2[i][1],0]) cube(cube_size);}


List comprehensions in OpenSCAD are, from what I understand, limited to taking a list as input, performing an operation on the individual elements (and even doing the operation may be conditional depending on the contents of the element), and returning a list. "Actions" can't be performed such as translate(); and cube();, but I can perform math on the elements and return a list of modified elements (e.g., convert from polar to cartesian coordinates).

List comprehensions seem to be implemented in such a way that they are not limited to 10,000 element lists (I don't see any errors when I input lists bigger than that in this example code), so it must have been written in the back-end to not create a huge stack I assume. I don't mind switching to this syntax for operations on lists - it is no doubt more efficient - but on the end of a) creating, and b) doing something with lists that large after the math is done, there doesn't seem to be a way to do that, since both would require "action" syntax that I can only get to work in a normal for() loop. Is there a way around this with a better use of OpenSCAD's language features that I just don't understand well enough yet, or am I stuck with batching the for() loop calls to sets with less than 10,000 elements?

Thanks,
David








_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org



_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: for() loop iteration limitation & list comprehensions

jon_bondy
Why not make the limit configurable, with a default of 10,000?

Jon

On 3/8/2017 4:50 PM, nop head wrote:

> The for loop limit is not really anything to do with stack use, that
> would be a recursion limit. I think it is just an arbitrary limit to
> catch infinite loops or massive trees of objects that will cause CGAL
> to explode in time and or memory use. A list on the other hand is a
> much smaller data structure than a CGAL object.
>
> If you have a practical use for such massive trees of objects then
> perhaps the limit can be increased. How many do you need and are you
> going to need to F6 the result? F5 might be OK but F6 with 10000
> objects doesn't seem practical.


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: for() loop iteration limitation & list comprehensions

nophead
Only any point if there is a practical use for 10000+ objects.

On 8 March 2017 at 22:00, jon <[hidden email]> wrote:
Why not make the limit configurable, with a default of 10,000?

Jon

On 3/8/2017 4:50 PM, nop head wrote:
The for loop limit is not really anything to do with stack use, that would be a recursion limit. I think it is just an arbitrary limit to catch infinite loops or massive trees of objects that will cause CGAL to explode in time and or memory use. A list on the other hand is a much smaller data structure than a CGAL object.

If you have a practical use for such massive trees of objects then perhaps the limit can be increased. How many do you need and are you going to need to F6 the result? F5 might be OK but F6 with 10000 objects doesn't seem practical.


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: for() loop iteration limitation & list comprehensions

davidconeff
In reply to this post by nophead
For cases where you have a lot of objects overlapping each other (say you generate 1000s of cubes lying on the same unit circle), CGAL will sieze up trying to render maybe the first 1000 objects. In the case of, say, 10000 cubes lying on a unit sphere evenly-spaced grid, CGAL doesn't have as much of a problem although it does take a while to render. The main issue where I see things start to lag is when you try to render objects that overlap in a lot of areas and it has to compute the end result inner surface or combined outer surface of many objects. So if you are careful about how much intersection happens (e.g., by creating a wire-frame grid where there are regular intersections only at the ends of the wires), CGAL can potentially render a lot more than 10000 objects in one call. 

On Wed, Mar 8, 2017 at 2:50 PM, nop head <[hidden email]> wrote:
The for loop limit is not really anything to do with stack use, that would be a recursion limit. I think it is just an arbitrary limit to catch infinite loops or massive trees of objects that will cause CGAL to explode in time and or memory use. A list on the other hand is a much smaller data structure than a CGAL object.

If you have a practical use for such massive trees of objects then perhaps the limit can be increased. How many do you need and are you going to need to F6 the result? F5 might be OK but F6 with 10000 objects doesn't seem practical.



On 8 March 2017 at 19:47, David Coneff <[hidden email]> wrote:
I am attempting to figure out how to do operations with "large" sets (>10,000 elements), which presents a couple problems in OpenSCAD that I believe there is a solution to, but I can't quite figure it out. I posted earlier about issues with for() being limited to an iteration range of 10,000, looking back this seems like a reasonable limitation since it points out that you're probably doing something inefficient that should otherwise be done with list comprehensions or something that is not stack-intensive.

Example code:

function generate_random_polar(n=5001, r=1)=[for (i=[0:1:n-1]) [r, rands(0,360,1)[0]]];
function polar_to_xy(r_theta=[1,0]) = [r_theta[0]*cos(r_theta[1]),r_theta[0]*sin(r_theta[1])];
list1 = concat(generate_random_polar(),generate_random_polar());
list2 = [for (x = list1) polar_to_xy(x)];

Problem 1: Algorithmic generation of a list with greater than 10,000 elements

The only way I've figured out how to do this is simple concatenation of two more or more smaller lists that are generated by a function call with <10,000 iterations in the for() loop. For lists of random points, no big deal. If I was trying to generate a set of ordered points with certain spaces in a certain canonical order, this could get more complicated to have to "batch" the function calls. Is there a better way?

Problem 2: Performing actions that require access to the inside of a list element (e.g., a coordinate pair)

Say I have a list of coordinates (x,y), and that list is greater than 10,000 elements. How would I loop over the entire list and perform an action such as plotting a cube at that point?, e.g.:

cube_size = 0.01;
for(i = [0:1:len(list2)-1]) {translate([list2[i][0],list2[i][1],0]) cube(cube_size);}


List comprehensions in OpenSCAD are, from what I understand, limited to taking a list as input, performing an operation on the individual elements (and even doing the operation may be conditional depending on the contents of the element), and returning a list. "Actions" can't be performed such as translate(); and cube();, but I can perform math on the elements and return a list of modified elements (e.g., convert from polar to cartesian coordinates).

List comprehensions seem to be implemented in such a way that they are not limited to 10,000 element lists (I don't see any errors when I input lists bigger than that in this example code), so it must have been written in the back-end to not create a huge stack I assume. I don't mind switching to this syntax for operations on lists - it is no doubt more efficient - but on the end of a) creating, and b) doing something with lists that large after the math is done, there doesn't seem to be a way to do that, since both would require "action" syntax that I can only get to work in a normal for() loop. Is there a way around this with a better use of OpenSCAD's language features that I just don't understand well enough yet, or am I stuck with batching the for() loop calls to sets with less than 10,000 elements?

Thanks,
David








_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org



_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org



_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: for() loop iteration limitation & list comprehensions

nophead
The trouble is OpenSCAD doesn't know which scenerio it is. 10000 might be far too many overlapping objects, or not enough when they don't overlap.

It seems like too blunt a tool, so perhaps the limit should simply go and the user gets what they deserve.

On 8 March 2017 at 22:05, David Coneff <[hidden email]> wrote:
For cases where you have a lot of objects overlapping each other (say you generate 1000s of cubes lying on the same unit circle), CGAL will sieze up trying to render maybe the first 1000 objects. In the case of, say, 10000 cubes lying on a unit sphere evenly-spaced grid, CGAL doesn't have as much of a problem although it does take a while to render. The main issue where I see things start to lag is when you try to render objects that overlap in a lot of areas and it has to compute the end result inner surface or combined outer surface of many objects. So if you are careful about how much intersection happens (e.g., by creating a wire-frame grid where there are regular intersections only at the ends of the wires), CGAL can potentially render a lot more than 10000 objects in one call. 

On Wed, Mar 8, 2017 at 2:50 PM, nop head <[hidden email]> wrote:
The for loop limit is not really anything to do with stack use, that would be a recursion limit. I think it is just an arbitrary limit to catch infinite loops or massive trees of objects that will cause CGAL to explode in time and or memory use. A list on the other hand is a much smaller data structure than a CGAL object.

If you have a practical use for such massive trees of objects then perhaps the limit can be increased. How many do you need and are you going to need to F6 the result? F5 might be OK but F6 with 10000 objects doesn't seem practical.



On 8 March 2017 at 19:47, David Coneff <[hidden email]> wrote:
I am attempting to figure out how to do operations with "large" sets (>10,000 elements), which presents a couple problems in OpenSCAD that I believe there is a solution to, but I can't quite figure it out. I posted earlier about issues with for() being limited to an iteration range of 10,000, looking back this seems like a reasonable limitation since it points out that you're probably doing something inefficient that should otherwise be done with list comprehensions or something that is not stack-intensive.

Example code:

function generate_random_polar(n=5001, r=1)=[for (i=[0:1:n-1]) [r, rands(0,360,1)[0]]];
function polar_to_xy(r_theta=[1,0]) = [r_theta[0]*cos(r_theta[1]),r_theta[0]*sin(r_theta[1])];
list1 = concat(generate_random_polar(),generate_random_polar());
list2 = [for (x = list1) polar_to_xy(x)];

Problem 1: Algorithmic generation of a list with greater than 10,000 elements

The only way I've figured out how to do this is simple concatenation of two more or more smaller lists that are generated by a function call with <10,000 iterations in the for() loop. For lists of random points, no big deal. If I was trying to generate a set of ordered points with certain spaces in a certain canonical order, this could get more complicated to have to "batch" the function calls. Is there a better way?

Problem 2: Performing actions that require access to the inside of a list element (e.g., a coordinate pair)

Say I have a list of coordinates (x,y), and that list is greater than 10,000 elements. How would I loop over the entire list and perform an action such as plotting a cube at that point?, e.g.:

cube_size = 0.01;
for(i = [0:1:len(list2)-1]) {translate([list2[i][0],list2[i][1],0]) cube(cube_size);}


List comprehensions in OpenSCAD are, from what I understand, limited to taking a list as input, performing an operation on the individual elements (and even doing the operation may be conditional depending on the contents of the element), and returning a list. "Actions" can't be performed such as translate(); and cube();, but I can perform math on the elements and return a list of modified elements (e.g., convert from polar to cartesian coordinates).

List comprehensions seem to be implemented in such a way that they are not limited to 10,000 element lists (I don't see any errors when I input lists bigger than that in this example code), so it must have been written in the back-end to not create a huge stack I assume. I don't mind switching to this syntax for operations on lists - it is no doubt more efficient - but on the end of a) creating, and b) doing something with lists that large after the math is done, there doesn't seem to be a way to do that, since both would require "action" syntax that I can only get to work in a normal for() loop. Is there a way around this with a better use of OpenSCAD's language features that I just don't understand well enough yet, or am I stuck with batching the for() loop calls to sets with less than 10,000 elements?

Thanks,
David








_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org



_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org



_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org



_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: for() loop iteration limitation & list comprehensions

Ronaldo
In reply to this post by davidconeff
David,

The main aspect to understand is that there is a for limit just for ranges.

If you don't use range in a for, there is no limit.

p = rands(0,1,1000000); // p is a list of 1000000 elements
for(pi=p){ // there is no range here
    if(pi<1e-6) echo(pi=pi);
}


Or if all your ranges have less than 10000 elements, you will not get an error message.

// a very long for with ranges
for(i=[0:999], j=[0:999], k=1000*i+j){
     if(k<10 ||  k> 999990) echo(k=k);
}

The same happens with for in list comprehension:

// short lists filtered from a long lists
q = [for(pi=p) if(pi<1e-6) pi];
s = [for(i=[0:999], j=[0:999], k=1000*i+j) if(k<len(p)-1 && p[k]<1e-6 ) p[k] ];
echo(q=q);
echo(s=s);


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: for() loop iteration limitation & list comprehensions

Ronaldo
In reply to this post by davidconeff
davidconeff wrote
 In the case of, say, 10000
cubes lying on a unit sphere evenly-spaced grid, CGAL doesn't have as much
of a problem although it does take a while to render.
Are you sure of that? As far as I know neither OpenSCAD nor CGAL use bounding boxes.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: for() loop iteration limitation & list comprehensions

nophead
for(x = [0:99], y = [0:99])
    translate([x * 10, y * 10, 0])
        sphere(1);

Takes 1 hour 18 minutes on my system and uses 2.6GB of memory. So perhaps 10K is a bit low, but not much before it would bring my system to its knees as Windows doesn't handle running out of memory very well.



On 8 March 2017 at 23:02, Ronaldo <[hidden email]> wrote:
davidconeff wrote
>  In the case of, say, 10000
> cubes lying on a unit sphere evenly-spaced grid, CGAL doesn't have as much
> of a problem although it does take a while to render.

Are you sure of that? As far as I know neither OpenSCAD nor CGAL use
bounding boxes.



--
View this message in context: http://forum.openscad.org/for-loop-iteration-limitation-list-comprehensions-tp20806p20813.html
Sent from the OpenSCAD mailing list archive at Nabble.com.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: for() loop iteration limitation & list comprehensions

davidconeff
In reply to this post by Ronaldo
Ronaldo,

p = rands(0,1,1000000); // p is a list of 1000000 elements
for(pi=p){ // there is no range here
    if(pi<1e-6) echo(pi=pi);
}

Thanks for that example, it's exactly what I was looking for. I hadn't seen that syntax case used in the for() argument before, it looks like that just knows to go over the entire list in canonical order without having to specify a range and iteration interval. The more common (0:1:len(list)-1) is what I see in most other people's code, but this is a really cumbersome way to do it, and introduces that range limitation on a for() loop unnecessarily.

As for the unit sphere example, I have some code that can produce that type of list. From what I have observed in test-case code I've written, it has nothing to do with bounding box calculations - the CSG rendering does a quick and dirty render that doesn't complain about intersecting surfaces because it is also not trying to be precise. CGAL seems to have awareness of when primitives intersect and tries to produce one solid geometry that is a simplification of all the intersecting volumes, so if you create something like the thread of a screw using progressively rotated and translated rectangular cubes (someone else's library that I played with a little), then the result is a very slow render because it has to figure out how many of the vertices of the intersecting volumes are inside the end-result shape in order to discard them.

I found that creating a thread by first starting with a 2D primitive and going through a linear_extrusion(rotation) operation was much more efficient both in CSG and CGAL since it is generating the end-result vertices in the first place rather than having to do loads of extra 3D math. I would assume that if you also just limit the number of objects that intersect in a given region to a small number, the math is probably much simpler for CGAL as well. I won't know until I have a fully fleshed out test-framework, I'm fairly close to being there.



On Wed, Mar 8, 2017 at 4:02 PM, Ronaldo <[hidden email]> wrote:
davidconeff wrote
>  In the case of, say, 10000
> cubes lying on a unit sphere evenly-spaced grid, CGAL doesn't have as much
> of a problem although it does take a while to render.

Are you sure of that? As far as I know neither OpenSCAD nor CGAL use
bounding boxes.



--
View this message in context: http://forum.openscad.org/for-loop-iteration-limitation-list-comprehensions-tp20806p20813.html
Sent from the OpenSCAD mailing list archive at Nabble.com.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: for() loop iteration limitation & list comprehensions

davidconeff
In reply to this post by nophead
I have 16 GB of RAM, a 1 TB SSD, and a quad-core i7 in my laptop. The unfortunate thing about most software is that it is written to keep programs in check for lower spec systems, but frequently doesn't have the option to disable those limits or up them when you do have better hardware. I think it is perfectly reasonable to have a default limit of 10,000 since not everyone has the setup I do, it'd just be nice to have an option to change it if I want to do something dumb on purpose :).

On Wed, Mar 8, 2017 at 4:36 PM, nop head <[hidden email]> wrote:
for(x = [0:99], y = [0:99])
    translate([x * 10, y * 10, 0])
        sphere(1);

Takes 1 hour 18 minutes on my system and uses 2.6GB of memory. So perhaps 10K is a bit low, but not much before it would bring my system to its knees as Windows doesn't handle running out of memory very well.



On 8 March 2017 at 23:02, Ronaldo <[hidden email]> wrote:
davidconeff wrote
>  In the case of, say, 10000
> cubes lying on a unit sphere evenly-spaced grid, CGAL doesn't have as much
> of a problem although it does take a while to render.

Are you sure of that? As far as I know neither OpenSCAD nor CGAL use
bounding boxes.



--
View this message in context: http://forum.openscad.org/for-loop-iteration-limitation-list-comprehensions-tp20806p20813.html
Sent from the OpenSCAD mailing list archive at Nabble.com.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org



_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: for() loop iteration limitation & list comprehensions

davidconeff
Actually, just realized 24 GB of RAM. 16 GB is available after I have a bunch of programs running and a few different OpenSCAD instances.

On Wed, Mar 8, 2017 at 4:47 PM, David Coneff <[hidden email]> wrote:
I have 16 GB of RAM, a 1 TB SSD, and a quad-core i7 in my laptop. The unfortunate thing about most software is that it is written to keep programs in check for lower spec systems, but frequently doesn't have the option to disable those limits or up them when you do have better hardware. I think it is perfectly reasonable to have a default limit of 10,000 since not everyone has the setup I do, it'd just be nice to have an option to change it if I want to do something dumb on purpose :).

On Wed, Mar 8, 2017 at 4:36 PM, nop head <[hidden email]> wrote:
for(x = [0:99], y = [0:99])
    translate([x * 10, y * 10, 0])
        sphere(1);

Takes 1 hour 18 minutes on my system and uses 2.6GB of memory. So perhaps 10K is a bit low, but not much before it would bring my system to its knees as Windows doesn't handle running out of memory very well.



On 8 March 2017 at 23:02, Ronaldo <[hidden email]> wrote:
davidconeff wrote
>  In the case of, say, 10000
> cubes lying on a unit sphere evenly-spaced grid, CGAL doesn't have as much
> of a problem although it does take a while to render.

Are you sure of that? As far as I know neither OpenSCAD nor CGAL use
bounding boxes.



--
View this message in context: http://forum.openscad.org/for-loop-iteration-limitation-list-comprehensions-tp20806p20813.html
Sent from the OpenSCAD mailing list archive at Nabble.com.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org




_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: for() loop iteration limitation & list comprehensions

nophead
In reply to this post by davidconeff
CGAL is extremely slow when there are a large number of non overlapping facets. It gets even slower if they do overlap but it certainly doesn't skip the easy cases.


On 8 March 2017 at 23:47, David Coneff <[hidden email]> wrote:
I have 16 GB of RAM, a 1 TB SSD, and a quad-core i7 in my laptop. The unfortunate thing about most software is that it is written to keep programs in check for lower spec systems, but frequently doesn't have the option to disable those limits or up them when you do have better hardware. I think it is perfectly reasonable to have a default limit of 10,000 since not everyone has the setup I do, it'd just be nice to have an option to change it if I want to do something dumb on purpose :).

On Wed, Mar 8, 2017 at 4:36 PM, nop head <[hidden email]> wrote:
for(x = [0:99], y = [0:99])
    translate([x * 10, y * 10, 0])
        sphere(1);

Takes 1 hour 18 minutes on my system and uses 2.6GB of memory. So perhaps 10K is a bit low, but not much before it would bring my system to its knees as Windows doesn't handle running out of memory very well.



On 8 March 2017 at 23:02, Ronaldo <[hidden email]> wrote:
davidconeff wrote
>  In the case of, say, 10000
> cubes lying on a unit sphere evenly-spaced grid, CGAL doesn't have as much
> of a problem although it does take a while to render.

Are you sure of that? As far as I know neither OpenSCAD nor CGAL use
bounding boxes.



--
View this message in context: http://forum.openscad.org/for-loop-iteration-limitation-list-comprehensions-tp20806p20813.html
Sent from the OpenSCAD mailing list archive at Nabble.com.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org



_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org



_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: for() loop iteration limitation & list comprehensions

davidconeff
I agree it does get slow. Just that it won't necessarily seize up or throw an error simply because there are a bunch. But that's to be expected for any model of sufficient complexity.

On Wed, Mar 8, 2017 at 4:54 PM, nop head <[hidden email]> wrote:
CGAL is extremely slow when there are a large number of non overlapping facets. It gets even slower if they do overlap but it certainly doesn't skip the easy cases.


On 8 March 2017 at 23:47, David Coneff <[hidden email]> wrote:
I have 16 GB of RAM, a 1 TB SSD, and a quad-core i7 in my laptop. The unfortunate thing about most software is that it is written to keep programs in check for lower spec systems, but frequently doesn't have the option to disable those limits or up them when you do have better hardware. I think it is perfectly reasonable to have a default limit of 10,000 since not everyone has the setup I do, it'd just be nice to have an option to change it if I want to do something dumb on purpose :).

On Wed, Mar 8, 2017 at 4:36 PM, nop head <[hidden email]> wrote:
for(x = [0:99], y = [0:99])
    translate([x * 10, y * 10, 0])
        sphere(1);

Takes 1 hour 18 minutes on my system and uses 2.6GB of memory. So perhaps 10K is a bit low, but not much before it would bring my system to its knees as Windows doesn't handle running out of memory very well.



On 8 March 2017 at 23:02, Ronaldo <[hidden email]> wrote:
davidconeff wrote
>  In the case of, say, 10000
> cubes lying on a unit sphere evenly-spaced grid, CGAL doesn't have as much
> of a problem although it does take a while to render.

Are you sure of that? As far as I know neither OpenSCAD nor CGAL use
bounding boxes.



--
View this message in context: http://forum.openscad.org/for-loop-iteration-limitation-list-comprehensions-tp20806p20813.html
Sent from the OpenSCAD mailing list archive at Nabble.com.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org



_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org



_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org



_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: for() loop iteration limitation & list comprehensions

MichaelAtOz
Administrator
I've had some things which rendered in 24+ hours.
(slowed by Windows swapping to SSD)
If someone wants to make some normally unreasonable thing, it is their time they are potentially wasting.

I think a limit should be an advanced option, AND, it should probably apply to the other cases Ronald highlighted too.
Admin - PM me if you need anything,
or if I've done something stupid...

Unless specifically shown otherwise above, my contribution is in the Public Domain; to the extent possible under law, I have waived all copyright and related or neighbouring rights to this work.
Obviously inclusion of works of previous authors is not included in the above.


The TPP is no simple “trade agreement.” Fight it! http://www.ourfairdeal.org/ time is running out!
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: for() loop iteration limitation & list comprehensions

Ronaldo
I see the limit of ranges, lists, whatever as a constraint. It may seem reasonable to avoid a long unwanted run by a beginner. But it may be a bottleneck for many serious work. So, David proposal that the limit may be extended by the user is a sensible one.

It may seem reasonable to extend the guardianship to lists. But a bi-dimensional matrix of 100x100 has 10000 elements. Should it be prohibited? I have implemented some f-rep modelling methods in OpenSCAD. Often that system build tri-dimensional matrices greater than 100x100x100 and build lists of triangles with more than 20000 elements. Any organic model will have this sort of size. Should all this be forbidden because this is not the intended application?

OpenSCAD is a powerful tool and should not be restricted to the beginner that builds models with a handful of cubes. Or it will be no more than a hammer, and worst, a hammer toy.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: for() loop iteration limitation & list comprehensions

davidconeff
That's my general feeling Ronaldo, I started playing with OpenSCAD because it is open-source and free, has a wide user base in the 3D printing community, and readily translates models on a computer to valid STL files that can be printed without much fuss if the appropriate level of detail is applied to the CGAL render. There's no reason it can't be used for serious work, it just has a less developed UI than the average piece of commercial software. For my purposes that is fine since the work I'm doing with is generative/algorithmic and not stuff that could be done by hand with a cursor in a well-developed GUI. It's just the minor limitations that I run into that are occasionally annoying, like this 10,000 element limitation. It's reasonable to prevent beginners from blowing up the program and thinking it is unstable on their mediocre computers (it's remarkably solid for open-source!), and to nudge you in the direction of using more sophisticated language features where appropriate (e.g., list comprehensions for purely mathematical stuff), but there are plenty of situations where that hard-coded limit is just an unnecessary obstacle. Sometimes you need to be able to use an inefficient script, see if the concept works, and refactor later.

On Wed, Mar 8, 2017 at 8:01 PM, Ronaldo Persiano <[hidden email]> wrote:
I see the limit of ranges, lists, whatever as a constraint. It may seem reasonable to avoid a long unwanted run by a beginner. But it may be a bottleneck for many serious work. So, David proposal that the limit may be extended by the user is a sensible one.

It may seem reasonable to extend the guardianship to lists. But a bi-dimensional matrix of 100x100 has 10000 elements. Should it be prohibited? I have implemented some f-rep modelling methods in OpenSCAD. Often that system build tri-dimensional matrices greater than 100x100x100 and build lists of triangles with more than 20000 elements. Any organic model will have this sort of size. Should all this be forbidden because this is not the intended application?

OpenSCAD is a powerful tool and should not be restricted to the beginner that builds models with a handful of cubes. Or it will be no more than a hammer, and worst, a hammer toy.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org



_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: for() loop iteration limitation & list comprehensions

kintel
Administrator
I agree that there are valid workflows outside our hardcoded limits.
Does anyone want to take a shot at refactoring the limit system?
I’d see a few relatively isolated steps:
o Make it configurable (using our config file)
o Make it adjustable using GUI/Preferences
o Detect limit situations and offer helpful dialogs offering users to cancel / ignore limit once / Never ask me again

The primary challenge is that once we start certain operations, we cannot cancel. When we run our own code, this is fixable, but if we hit a non-cancellable entry point to a 3rd party library it gets trickier (e.g. CGAL).

 -Marius


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: for() loop iteration limitation & list comprehensions

davidconeff
In reply to this post by Ronaldo
Ronaldo,

Since you knew the for() syntax that bypasses the 10,000 element limitation, I was wondering if you might also know the solution to this:

Say I have a list of the form:

[[x,y,z],[x2,y2,z2],....]

How would I make a for() loop that performs an action using the (i)-th and (i+1)th elements? Normally I would use the iterator variable, but the syntax that bypasses the 10,000 range limit doesn't have "access" to an iterator variable.

simple example of what I would normally do:

points = [[1,1,1],[2,2,2],[3,3,3]]
for(i = [0:1:len(points-2)]){
     translate([points[i][0],points[i+1][1],0])cube(1);
}

On Wed, Mar 8, 2017 at 3:54 PM, Ronaldo Persiano <[hidden email]> wrote:
David,

The main aspect to understand is that there is a for limit just for ranges.

If you don't use range in a for, there is no limit.

p = rands(0,1,1000000); // p is a list of 1000000 elements
for(pi=p){ // there is no range here
    if(pi<1e-6) echo(pi=pi);
}


Or if all your ranges have less than 10000 elements, you will not get an error message.

// a very long for with ranges
for(i=[0:999], j=[0:999], k=1000*i+j){
     if(k<10 ||  k> 999990) echo(k=k);
}

The same happens with for in list comprehension:

// short lists filtered from a long lists
q = [for(pi=p) if(pi<1e-6) pi];
s = [for(i=[0:999], j=[0:999], k=1000*i+j) if(k<len(p)-1 && p[k]<1e-6 ) p[k] ];
echo(q=q);
echo(s=s);


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org



_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: for() loop iteration limitation & list comprehensions

nophead
I think this a work around that will allow a for loop of 10,000,000

module for_n(n) {
    thousands = floor(n / 1000);
    remainder = n % 1000;
    if(thousands)
        for(a = [0 : thousands - 1], b = [0 : 999], $i = a * 1000 + b)
            children();
    if(remainder)
        for(a = [0 : remainder - 1], $i = thousands * 1000 + a)
            children();
}
   
for_n(1002)
        echo($i);

Note that it counts from 0 to n-1

On 9 March 2017 at 04:41, David Coneff <[hidden email]> wrote:
Ronaldo,

Since you knew the for() syntax that bypasses the 10,000 element limitation, I was wondering if you might also know the solution to this:

Say I have a list of the form:

[[x,y,z],[x2,y2,z2],....]

How would I make a for() loop that performs an action using the (i)-th and (i+1)th elements? Normally I would use the iterator variable, but the syntax that bypasses the 10,000 range limit doesn't have "access" to an iterator variable.

simple example of what I would normally do:

points = [[1,1,1],[2,2,2],[3,3,3]]
for(i = [0:1:len(points-2)]){
     translate([points[i][0],points[i+1][1],0])cube(1);
}

On Wed, Mar 8, 2017 at 3:54 PM, Ronaldo Persiano <[hidden email]> wrote:
David,

The main aspect to understand is that there is a for limit just for ranges.

If you don't use range in a for, there is no limit.

p = rands(0,1,1000000); // p is a list of 1000000 elements
for(pi=p){ // there is no range here
    if(pi<1e-6) echo(pi=pi);
}


Or if all your ranges have less than 10000 elements, you will not get an error message.

// a very long for with ranges
for(i=[0:999], j=[0:999], k=1000*i+j){
     if(k<10 ||  k> 999990) echo(k=k);
}

The same happens with for in list comprehension:

// short lists filtered from a long lists
q = [for(pi=p) if(pi<1e-6) pi];
s = [for(i=[0:999], j=[0:999], k=1000*i+j) if(k<len(p)-1 && p[k]<1e-6 ) p[k] ];
echo(q=q);
echo(s=s);


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org



_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org



_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
12
Loading...