# for() loop iteration limitation & list comprehensions

28 messages
12
Open this post in threaded view
|
Report Content as Inappropriate

## for() loop iteration limitation & list comprehensions

 I am attempting to figure out how to do operations with "large" sets (>10,000 elements), which presents a couple problems in OpenSCAD that I believe there is a solution to, but I can't quite figure it out. I posted earlier about issues with for() being limited to an iteration range of 10,000, looking back this seems like a reasonable limitation since it points out that you're probably doing something inefficient that should otherwise be done with list comprehensions or something that is not stack-intensive.Example code:function generate_random_polar(n=5001, r=1)=[for (i=[0:1:n-1]) [r, rands(0,360,1)[0]]];function polar_to_xy(r_theta=[1,0]) = [r_theta[0]*cos(r_theta[1]),r_theta[0]*sin(r_theta[1])];list1 = concat(generate_random_polar(),generate_random_polar());list2 = [for (x = list1) polar_to_xy(x)];Problem 1: Algorithmic generation of a list with greater than 10,000 elementsThe only way I've figured out how to do this is simple concatenation of two more or more smaller lists that are generated by a function call with <10,000 iterations in the for() loop. For lists of random points, no big deal. If I was trying to generate a set of ordered points with certain spaces in a certain canonical order, this could get more complicated to have to "batch" the function calls. Is there a better way?Problem 2: Performing actions that require access to the inside of a list element (e.g., a coordinate pair)Say I have a list of coordinates (x,y), and that list is greater than 10,000 elements. How would I loop over the entire list and perform an action such as plotting a cube at that point?, e.g.:cube_size = 0.01;for(i = [0:1:len(list2)-1]) {translate([list2[i][0],list2[i][1],0]) cube(cube_size);}List comprehensions in OpenSCAD are, from what I understand, limited to taking a list as input, performing an operation on the individual elements (and even doing the operation may be conditional depending on the contents of the element), and returning a list. "Actions" can't be performed such as translate(); and cube();, but I can perform math on the elements and return a list of modified elements (e.g., convert from polar to cartesian coordinates).List comprehensions seem to be implemented in such a way that they are not limited to 10,000 element lists (I don't see any errors when I input lists bigger than that in this example code), so it must have been written in the back-end to not create a huge stack I assume. I don't mind switching to this syntax for operations on lists - it is no doubt more efficient - but on the end of a) creating, and b) doing something with lists that large after the math is done, there doesn't seem to be a way to do that, since both would require "action" syntax that I can only get to work in a normal for() loop. Is there a way around this with a better use of OpenSCAD's language features that I just don't understand well enough yet, or am I stuck with batching the for() loop calls to sets with less than 10,000 elements?Thanks,David _______________________________________________ OpenSCAD mailing list [hidden email] http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Open this post in threaded view
|
Report Content as Inappropriate

## Re: for() loop iteration limitation & list comprehensions

Open this post in threaded view
|
Report Content as Inappropriate

## Re: for() loop iteration limitation & list comprehensions

 Why not make the limit configurable, with a default of 10,000? Jon On 3/8/2017 4:50 PM, nop head wrote: > The for loop limit is not really anything to do with stack use, that > would be a recursion limit. I think it is just an arbitrary limit to > catch infinite loops or massive trees of objects that will cause CGAL > to explode in time and or memory use. A list on the other hand is a > much smaller data structure than a CGAL object. > > If you have a practical use for such massive trees of objects then > perhaps the limit can be increased. How many do you need and are you > going to need to F6 the result? F5 might be OK but F6 with 10000 > objects doesn't seem practical. _______________________________________________ OpenSCAD mailing list [hidden email] http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Open this post in threaded view
|
Report Content as Inappropriate

## Re: for() loop iteration limitation & list comprehensions

 Only any point if there is a practical use for 10000+ objects.On 8 March 2017 at 22:00, jon wrote:Why not make the limit configurable, with a default of 10,000? Jon On 3/8/2017 4:50 PM, nop head wrote: The for loop limit is not really anything to do with stack use, that would be a recursion limit. I think it is just an arbitrary limit to catch infinite loops or massive trees of objects that will cause CGAL to explode in time and or memory use. A list on the other hand is a much smaller data structure than a CGAL object. If you have a practical use for such massive trees of objects then perhaps the limit can be increased. How many do you need and are you going to need to F6 the result? F5 might be OK but F6 with 10000 objects doesn't seem practical. _______________________________________________ OpenSCAD mailing list [hidden email] http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org _______________________________________________ OpenSCAD mailing list [hidden email] http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Open this post in threaded view
|
Report Content as Inappropriate

## Re: for() loop iteration limitation & list comprehensions

Open this post in threaded view
|
Report Content as Inappropriate

## Re: for() loop iteration limitation & list comprehensions

Open this post in threaded view
|
Report Content as Inappropriate

## Re: for() loop iteration limitation & list comprehensions

 In reply to this post by davidconeff David,The main aspect to understand is that there is a for limit just for ranges. If you don't use range in a for, there is no limit.p = rands(0,1,1000000); // p is a list of 1000000 elementsfor(pi=p){ // there is no range here    if(pi<1e-6) echo(pi=pi);}Or if all your ranges have less than 10000 elements, you will not get an error message.// a very long for with rangesfor(i=[0:999], j=[0:999], k=1000*i+j){     if(k<10 ||  k> 999990) echo(k=k);}The same happens with for in list comprehension:// short lists filtered from a long listsq = [for(pi=p) if(pi<1e-6) pi];s = [for(i=[0:999], j=[0:999], k=1000*i+j) if(k
Open this post in threaded view
|
Report Content as Inappropriate

## Re: for() loop iteration limitation & list comprehensions

 In reply to this post by davidconeff davidconeff wrote  In the case of, say, 10000 cubes lying on a unit sphere evenly-spaced grid, CGAL doesn't have as much of a problem although it does take a while to render. Are you sure of that? As far as I know neither OpenSCAD nor CGAL use bounding boxes.
Open this post in threaded view
|
Report Content as Inappropriate

## Re: for() loop iteration limitation & list comprehensions

 for(x = [0:99], y = [0:99])    translate([x * 10, y * 10, 0])        sphere(1);Takes 1 hour 18 minutes on my system and uses 2.6GB of memory. So perhaps 10K is a bit low, but not much before it would bring my system to its knees as Windows doesn't handle running out of memory very well.On 8 March 2017 at 23:02, Ronaldo wrote:davidconeff wrote >  In the case of, say, 10000 > cubes lying on a unit sphere evenly-spaced grid, CGAL doesn't have as much > of a problem although it does take a while to render. Are you sure of that? As far as I know neither OpenSCAD nor CGAL use bounding boxes. -- View this message in context: http://forum.openscad.org/for-loop-iteration-limitation-list-comprehensions-tp20806p20813.html Sent from the OpenSCAD mailing list archive at Nabble.com. _______________________________________________ OpenSCAD mailing list [hidden email] http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org _______________________________________________ OpenSCAD mailing list [hidden email] http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Open this post in threaded view
|
Report Content as Inappropriate

## Re: for() loop iteration limitation & list comprehensions

Open this post in threaded view
|
Report Content as Inappropriate

## Re: for() loop iteration limitation & list comprehensions

Open this post in threaded view
|
Report Content as Inappropriate

## Re: for() loop iteration limitation & list comprehensions

Open this post in threaded view
|
Report Content as Inappropriate

## Re: for() loop iteration limitation & list comprehensions

Open this post in threaded view
|
Report Content as Inappropriate

## Re: for() loop iteration limitation & list comprehensions

Open this post in threaded view
|
Report Content as Inappropriate

## Re: for() loop iteration limitation & list comprehensions

 Administrator I've had some things which rendered in 24+ hours. (slowed by Windows swapping to SSD) If someone wants to make some normally unreasonable thing, it is their time they are potentially wasting. I think a limit should be an advanced option, AND, it should probably apply to the other cases Ronald highlighted too. Admin - PM me if you need anything, or if I've done something stupid... Unless specifically shown otherwise above, my contribution is in the Public Domain; to the extent possible under law, I have waived all copyright and related or neighbouring rights to this work. Obviously inclusion of works of previous authors is not included in the above. The TPP is no simple “trade agreement.” Fight it! http://www.ourfairdeal.org/ time is running out!
Open this post in threaded view
|
Report Content as Inappropriate

## Re: for() loop iteration limitation & list comprehensions

 I see the limit of ranges, lists, whatever as a constraint. It may seem reasonable to avoid a long unwanted run by a beginner. But it may be a bottleneck for many serious work. So, David proposal that the limit may be extended by the user is a sensible one. It may seem reasonable to extend the guardianship to lists. But a bi-dimensional matrix of 100x100 has 10000 elements. Should it be prohibited? I have implemented some f-rep modelling methods in OpenSCAD. Often that system build tri-dimensional matrices greater than 100x100x100 and build lists of triangles with more than 20000 elements. Any organic model will have this sort of size. Should all this be forbidden because this is not the intended application?OpenSCAD is a powerful tool and should not be restricted to the beginner that builds models with a handful of cubes. Or it will be no more than a hammer, and worst, a hammer toy. _______________________________________________ OpenSCAD mailing list [hidden email] http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Open this post in threaded view
|
Report Content as Inappropriate

## Re: for() loop iteration limitation & list comprehensions

 That's my general feeling Ronaldo, I started playing with OpenSCAD because it is open-source and free, has a wide user base in the 3D printing community, and readily translates models on a computer to valid STL files that can be printed without much fuss if the appropriate level of detail is applied to the CGAL render. There's no reason it can't be used for serious work, it just has a less developed UI than the average piece of commercial software. For my purposes that is fine since the work I'm doing with is generative/algorithmic and not stuff that could be done by hand with a cursor in a well-developed GUI. It's just the minor limitations that I run into that are occasionally annoying, like this 10,000 element limitation. It's reasonable to prevent beginners from blowing up the program and thinking it is unstable on their mediocre computers (it's remarkably solid for open-source!), and to nudge you in the direction of using more sophisticated language features where appropriate (e.g., list comprehensions for purely mathematical stuff), but there are plenty of situations where that hard-coded limit is just an unnecessary obstacle. Sometimes you need to be able to use an inefficient script, see if the concept works, and refactor later.On Wed, Mar 8, 2017 at 8:01 PM, Ronaldo Persiano wrote:I see the limit of ranges, lists, whatever as a constraint. It may seem reasonable to avoid a long unwanted run by a beginner. But it may be a bottleneck for many serious work. So, David proposal that the limit may be extended by the user is a sensible one. It may seem reasonable to extend the guardianship to lists. But a bi-dimensional matrix of 100x100 has 10000 elements. Should it be prohibited? I have implemented some f-rep modelling methods in OpenSCAD. Often that system build tri-dimensional matrices greater than 100x100x100 and build lists of triangles with more than 20000 elements. Any organic model will have this sort of size. Should all this be forbidden because this is not the intended application?OpenSCAD is a powerful tool and should not be restricted to the beginner that builds models with a handful of cubes. Or it will be no more than a hammer, and worst, a hammer toy. _______________________________________________ OpenSCAD mailing list [hidden email] http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org _______________________________________________ OpenSCAD mailing list [hidden email] http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Open this post in threaded view
|
Report Content as Inappropriate

## Re: for() loop iteration limitation & list comprehensions

 Administrator I agree that there are valid workflows outside our hardcoded limits. Does anyone want to take a shot at refactoring the limit system? I’d see a few relatively isolated steps: o Make it configurable (using our config file) o Make it adjustable using GUI/Preferences o Detect limit situations and offer helpful dialogs offering users to cancel / ignore limit once / Never ask me again The primary challenge is that once we start certain operations, we cannot cancel. When we run our own code, this is fixable, but if we hit a non-cancellable entry point to a 3rd party library it gets trickier (e.g. CGAL).  -Marius _______________________________________________ OpenSCAD mailing list [hidden email] http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Open this post in threaded view
|
Report Content as Inappropriate

## Re: for() loop iteration limitation & list comprehensions

 In reply to this post by Ronaldo Ronaldo,Since you knew the for() syntax that bypasses the 10,000 element limitation, I was wondering if you might also know the solution to this:Say I have a list of the form:[[x,y,z],[x2,y2,z2],....]How would I make a for() loop that performs an action using the (i)-th and (i+1)th elements? Normally I would use the iterator variable, but the syntax that bypasses the 10,000 range limit doesn't have "access" to an iterator variable.simple example of what I would normally do:points = [[1,1,1],[2,2,2],[3,3,3]]for(i = [0:1:len(points-2)]){     translate([points[i][0],points[i+1][1],0])cube(1);}On Wed, Mar 8, 2017 at 3:54 PM, Ronaldo Persiano wrote:David,The main aspect to understand is that there is a for limit just for ranges. If you don't use range in a for, there is no limit.p = rands(0,1,1000000); // p is a list of 1000000 elementsfor(pi=p){ // there is no range here    if(pi<1e-6) echo(pi=pi);}Or if all your ranges have less than 10000 elements, you will not get an error message.// a very long for with rangesfor(i=[0:999], j=[0:999], k=1000*i+j){     if(k<10 ||  k> 999990) echo(k=k);}The same happens with for in list comprehension:// short lists filtered from a long listsq = [for(pi=p) if(pi<1e-6) pi];s = [for(i=[0:999], j=[0:999], k=1000*i+j) if(k
Open this post in threaded view
|
Report Content as Inappropriate

## Re: for() loop iteration limitation & list comprehensions

 I think this a work around that will allow a for loop of 10,000,000module for_n(n) {    thousands = floor(n / 1000);    remainder = n % 1000;    if(thousands)        for(a = [0 : thousands - 1], b = [0 : 999], \$i = a * 1000 + b)            children();    if(remainder)        for(a = [0 : remainder - 1], \$i = thousands * 1000 + a)            children();}    for_n(1002)        echo(\$i);Note that it counts from 0 to n-1On 9 March 2017 at 04:41, David Coneff wrote:Ronaldo,Since you knew the for() syntax that bypasses the 10,000 element limitation, I was wondering if you might also know the solution to this:Say I have a list of the form:[[x,y,z],[x2,y2,z2],....]How would I make a for() loop that performs an action using the (i)-th and (i+1)th elements? Normally I would use the iterator variable, but the syntax that bypasses the 10,000 range limit doesn't have "access" to an iterator variable.simple example of what I would normally do:points = [[1,1,1],[2,2,2],[3,3,3]]for(i = [0:1:len(points-2)]){     translate([points[i][0],points[i+1][1],0])cube(1);}On Wed, Mar 8, 2017 at 3:54 PM, Ronaldo Persiano wrote:David,The main aspect to understand is that there is a for limit just for ranges. If you don't use range in a for, there is no limit.p = rands(0,1,1000000); // p is a list of 1000000 elementsfor(pi=p){ // there is no range here    if(pi<1e-6) echo(pi=pi);}Or if all your ranges have less than 10000 elements, you will not get an error message.// a very long for with rangesfor(i=[0:999], j=[0:999], k=1000*i+j){     if(k<10 ||  k> 999990) echo(k=k);}The same happens with for in list comprehension:// short lists filtered from a long listsq = [for(pi=p) if(pi<1e-6) pi];s = [for(i=[0:999], j=[0:999], k=1000*i+j) if(k
12