Memory usage.

classic Classic list List threaded Threaded
9 messages Options
rew
Reply | Threaded
Open this post in threaded view
|

Memory usage.

rew
Hi Guys,

I was watching the horrors of the memory use of Chrome, and the
developers there always say, but you need to look at RESident size:
actual use is a lot less. So chrome processes vary between 100 and
800Mb resident memory in use.

Then I saw an odd one out: Openscad: 1.8G ! (and anohter one at 1.0G).

Is it normal for openscad to use that much memory? I haven't touched
them for days...

(By the way: It's my OS's fault to have not swapped most of those
processes out, but getting THAT improved is not going to happen
anytime soon).

(one openscad may have a design loaded with 169 copies of a
button-cap).

        Roger.

--
** [hidden email] ** https://www.BitWizard.nl/ ** +31-15-2049110 **
**    Delftechpark 11 2628 XJ  Delft, The Netherlands.  KVK: 27239233    **
f equals m times a. When your f is steady, and your m is going down
your a is going up.  -- Chris Hadfield about flying up the space shuttle.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage.

Alan Cox-2
> Then I saw an odd one out: Openscad: 1.8G ! (and anohter one at 1.0G).
>
> Is it normal for openscad to use that much memory? I haven't touched
> them for days...

In my experience yes. OpenSCAD uses *lots* of memory. It also has a
deliberately included cache to speed up future operations. That (rightly)
isn't thrown away over time but will get swapped out if needed.
 
> (By the way: It's my OS's fault to have not swapped most of those
> processes out, but getting THAT improved is not going to happen
> anytime soon).

It's probbly not swapped that out because either

1. It's not under any memory pressure so hasn't bothered reclaiming it
2. The memory left in is shared stuff (C library, Qt, Boost, etc) with
other users
3. There's something causing things to be accessed in the background.
It's one of the fun things with a GUI app where you find stupid stuff like
a mouse pointer crossing into the window or a timer causing something to
walk a chunk of memory and keep it loaded, or that some of your memory is
actually pinned textures the GPU is rendering.

or some combination of the three.

What happens if you send it a SIGSTOP and then run other memory heavy
stuff to fill the system memory ?

Alan

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage.

nophead
In reply to this post by rew
Yes, OpenSCAD can use a lot of memory. If you keep changing a model old bits of the geometry are held in the cache. Sometimes I have to close and reopen it if I have been working on something for a few days.

On Thu, 8 Oct 2020 at 15:43, Rogier Wolff <[hidden email]> wrote:
Hi Guys,

I was watching the horrors of the memory use of Chrome, and the
developers there always say, but you need to look at RESident size:
actual use is a lot less. So chrome processes vary between 100 and
800Mb resident memory in use.

Then I saw an odd one out: Openscad: 1.8G ! (and anohter one at 1.0G).

Is it normal for openscad to use that much memory? I haven't touched
them for days...

(By the way: It's my OS's fault to have not swapped most of those
processes out, but getting THAT improved is not going to happen
anytime soon).

(one openscad may have a design loaded with 169 copies of a
button-cap).

        Roger.

--
** [hidden email] ** https://www.BitWizard.nl/ ** +31-15-2049110 **
**    Delftechpark 11 2628 XJ  Delft, The Netherlands.  KVK: 27239233    **
f equals m times a. When your f is steady, and your m is going down
your a is going up.  -- Chris Hadfield about flying up the space shuttle.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage.

JordanBrown
In reply to this post by Alan Cox-2
On 10/8/2020 8:42 AM, Alan Cox wrote:
In my experience yes. OpenSCAD uses *lots* of memory. It also has a deliberately included cache to speed up future operations. That (rightly) isn't thrown away over time but will get swapped out if needed.


Hey, internals people:  is the cache arranged so that the large stuff (the actual geometry) is kept separately from the smaller stuff (the metadata that you would match against to select a cache entry)?

I don't know how applicable that is, or how practical it is with typical memory allocators, but it seems like it would be useful.  You could have a gigabyte of geometry cached, but because you never *look* at it, because the metadata doesn't match what you're doing now, it could all be paged out.


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage.

MichaelAtOz
Administrator
You specify cache sizes in Preferences.
You can also use Design/Flush-caches.
You can see cache size in console after F5/F6.

I set my sizes big and let the OS page out if it needs to.



-----
OpenSCAD Admin - email* me if you need anything,  or if I've done something stupid...

* on the Forum, click on my MichaelAtOz label, there is a link to email me.

Unless specifically shown otherwise above, my contribution is in the Public Domain; to the extent possible under law, I have waived all copyright and related or neighbouring rights to this work. Obviously inclusion of works of previous authors is not included in the above.

--
Sent from: http://forum.openscad.org/

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
OpenSCAD Admin - email* me if you need anything, or if I've done something stupid...
* on the Forum, click on my MichaelAtOz label, there is a link to email me.

Unless specifically shown otherwise above, my contribution is in the Public Domain;
to the extent possible under law, I have waived all copyright and related or neighbouring rights to this work.
Obviously inclusion of works of previous authors is not included in the above.
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage.

MichaelAtOz
Administrator
MichaelAtOz wrote
> I set my sizes big and let the OS page out if it needs to.

But I have fast SSD's, when I was on HDD paging was painful.
Buy more memory!



-----
OpenSCAD Admin - email* me if you need anything,  or if I've done something stupid...

* on the Forum, click on my MichaelAtOz label, there is a link to email me.

Unless specifically shown otherwise above, my contribution is in the Public Domain; to the extent possible under law, I have waived all copyright and related or neighbouring rights to this work. Obviously inclusion of works of previous authors is not included in the above.

--
Sent from: http://forum.openscad.org/

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
OpenSCAD Admin - email* me if you need anything, or if I've done something stupid...
* on the Forum, click on my MichaelAtOz label, there is a link to email me.

Unless specifically shown otherwise above, my contribution is in the Public Domain;
to the extent possible under law, I have waived all copyright and related or neighbouring rights to this work.
Obviously inclusion of works of previous authors is not included in the above.
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage.

JordanBrown
In reply to this post by MichaelAtOz
On 10/8/2020 5:32 PM, MichaelAtOz wrote:
You specify cache sizes in Preferences.
You can also use Design/Flush-caches.
You can see cache size in console after F5/F6.

I set my sizes big and let the OS page out if it needs to.

Thanks, but doesn't answer the question.

I'm assuming that a cache entry consists of two parts:
(1) metadata that can be used to identify matching geometry.  Exactly what that looks like I don't know, but whatever it is that you use to index into the cache to find cached data.
(2) the actual cached data.  Triangle soup?

I'm assuming that the actual cached data is much larger than the index metadata.

Assuming that all of those assumptions are correct, a cache entry might be arranged in two different structures:

(a) a single memory allocation that contains both metadata and actual cached data.
(b) a memory allocation that contains metadata, and another that contains the actual cached data.

You want (b).  You don't want the metadata to be "near" the actual cached data.  If the metadata is "near" the actual cached data, then (maybe) merely searching for a cache hit will mark the page containing the cache data as recently accessed, and keep it from being paged out.

You want the actual cache data to be kept away from anything that gets touched on a routine basis, so that if it isn't actually used it's kept paged out.  If it's all mixed up in stuff that gets routinely touched, then it's more likely to get pulled into RAM and kept in RAM.

Another possible structure is that the cache data consists of a lot of small allocations, totally mixed in with other allocations.  That would probably be the worst of all worlds, because it would mean that your cache data would be "diluting" the stuff that you access all the time, keeping more pages in RAM than really need to be there.

Does that make sense now?


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage.

jotweh
There are two major caches in OpenSCAD, CGALCache f and GeometryCache  for Nef and non-Nef geometry respectively.
Both implement src/cache.h, mapping everything to a `std::unordered_map<std::string, struct with ptr to struct with shared_ptr<data> >`
There is no special allocator defined for these unordered_maps.

So to answer your Question:

It is more related yo tour strategy (b), but not entirely, because it is not all "in a single memory location", but structured as hash bucket map for fast cache lookups.
So it can be said: There is no explicit swap-friendly implementation of the general internal data structures.

If you want to help investigate if a better allocation strategy would help your general workflow, you can start your openscad with `tcmalloc`-preloads[1],[2] and tell us if it you can see any difference for your large projects.
tcmalloc has a better heap locality for similar sized objects and shows different heap usage, sometimes "better", maybe also in this case..

[1] Overview: https://google.github.io/tcmalloc/overview.html
[2] How to Preload: https://gperftools.github.io/gperftools/tcmalloc.html


On 09/10/2020 03.25, Jordan Brown wrote:
On 10/8/2020 5:32 PM, MichaelAtOz wrote:
You specify cache sizes in Preferences.
You can also use Design/Flush-caches.
You can see cache size in console after F5/F6.

I set my sizes big and let the OS page out if it needs to.

Thanks, but doesn't answer the question.

I'm assuming that a cache entry consists of two parts:
(1) metadata that can be used to identify matching geometry.  Exactly what that looks like I don't know, but whatever it is that you use to index into the cache to find cached data.
(2) the actual cached data.  Triangle soup?

I'm assuming that the actual cached data is much larger than the index metadata.

Assuming that all of those assumptions are correct, a cache entry might be arranged in two different structures:

(a) a single memory allocation that contains both metadata and actual cached data.
(b) a memory allocation that contains metadata, and another that contains the actual cached data.

You want (b).  You don't want the metadata to be "near" the actual cached data.  If the metadata is "near" the actual cached data, then (maybe) merely searching for a cache hit will mark the page containing the cache data as recently accessed, and keep it from being paged out.

You want the actual cache data to be kept away from anything that gets touched on a routine basis, so that if it isn't actually used it's kept paged out.  If it's all mixed up in stuff that gets routinely touched, then it's more likely to get pulled into RAM and kept in RAM.

Another possible structure is that the cache data consists of a lot of small allocations, totally mixed in with other allocations.  That would probably be the worst of all worlds, because it would mean that your cache data would be "diluting" the stuff that you access all the time, keeping more pages in RAM than really need to be there.

Does that make sense now?


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Memory usage.

OpenSCAD mailing list-2
In reply to this post by rew
I have a 32G workstation and I have definitely watched, nervously, while
openscad grew to 20G during a render.   Well, I have a lot of points in my
polygons...

I started out trying to fire up 3 or 4 instances of openscad to get faster
rendering, but the instances silently blew off when memory ran out.  Just
luckily I noticed how big they were, and I could only accommodate one at a
time!




-----
richalt2  (so why does the forum show me as OpenSCAD mailing list?)
Rich Altmaier

--
Sent from: http://forum.openscad.org/

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org