Auto Reload with lots of files causes massive slowdown again

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Auto Reload with lots of files causes massive slowdown again

nophead
I have fixed this problem twice but it is back again. My ability to debug it is severely limited now because I can no longer build at home. I trashed my MSYS environment by trying to add the double-conversion package.

Has anybody changed something recently that might have broken it? As far as I can tell the StatCache is working and only has one entry for each file but does everything still go through it?

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Auto Reload with lots of files causes massive slowdown again

kintel
Administrator
> On Feb 16, 2019, at 13:56, nop head <[hidden email]> wrote:
>
> I have fixed this problem twice but it is back again. My ability to debug it is severely limited now because I can no longer build at home. I trashed my MSYS environment by trying to add the double-conversion package.
>
> Has anybody changed something recently that might have broken it? As far as I can tell the StatCache is working and only has one entry for each file but does everything still go through it?

I cannot think of any recent breakage, but a lot of PRs came through recently, so we may have introduced a bug somewhere..

I still think that the correct solution to this is to use a filesystem monitor, which most filesystems, including NTFS, should support.

It would be good if we could get a local MSYS build up and running again. Just a severe lack of Windows developers around here : /
Looks like my XP VM finally died, so I'm out of luck at the moment as well.

A native (MSVC) build env would probably make life easier for anyone using Windows proper, but that's still quite a bit of work to pull off, mostly getting binaries to build for all dependencies.

 -Marius


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Auto Reload with lots of files causes massive slowdown again

nophead
Yes a filesystem monitor would be more efficient but multi-platform and on networked filesystems, etc?

Basically I have about 100 unique files and checking their dates doesn't take the five seconds I am seeing now. It should be a small fraction of a second. However lots of files are included many times so the original problem was they were being checked thousands of times. 

Somehow I think the problem is back and and what ever has broken it would probably still break a filesytem monitor solution. To take 5 seconds something must be accessing the files over and over again.

I only noticed the problem when I downloaded Hans' polygon speed up fork but I don't think that is the cause. It must have been another change to master. 

The obvious solution would be git bisect but my code won't work on old versions any more because the language keeps changing. It won't even work on the release candidate, which is disappointing because I was expecting to be able to release code that would work on the upcoming release,

Perhaps I can do git bisect using Mendel90 code as the test. However that would require getting MSYS2 to work again, or building a Linux box. I don't have long before I go overseas again with only a laptop.

On Sat, 16 Feb 2019 at 20:13, Marius Kintel <[hidden email]> wrote:
> On Feb 16, 2019, at 13:56, nop head <[hidden email]> wrote:
>
> I have fixed this problem twice but it is back again. My ability to debug it is severely limited now because I can no longer build at home. I trashed my MSYS environment by trying to add the double-conversion package.
>
> Has anybody changed something recently that might have broken it? As far as I can tell the StatCache is working and only has one entry for each file but does everything still go through it?

I cannot think of any recent breakage, but a lot of PRs came through recently, so we may have introduced a bug somewhere..

I still think that the correct solution to this is to use a filesystem monitor, which most filesystems, including NTFS, should support.

It would be good if we could get a local MSYS build up and running again. Just a severe lack of Windows developers around here : /
Looks like my XP VM finally died, so I'm out of luck at the moment as well.

A native (MSVC) build env would probably make life easier for anyone using Windows proper, but that's still quite a bit of work to pull off, mostly getting binaries to build for all dependencies.

 -Marius


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Auto Reload with lots of files causes massive slowdown again

kintel
Administrator
> On Feb 16, 2019, at 19:31, nop head <[hidden email]> wrote:
>
> The obvious solution would be git bisect but my code won't work on old versions any more because the language keeps changing. It won't even work on the release candidate, which is disappointing because I was expecting to be able to release code that would work on the upcoming release,
>
What did we change/not include that caused this to not work?

 -Marius


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Auto Reload with lots of files causes massive slowdown again

nophead
The warnings about len and undef and the introduction of is_indef and is_list as work arounds.

On Sun, 17 Feb 2019 at 03:20, Marius Kintel <[hidden email]> wrote:
> On Feb 16, 2019, at 19:31, nop head <[hidden email]> wrote:
>
> The obvious solution would be git bisect but my code won't work on old versions any more because the language keeps changing. It won't even work on the release candidate, which is disappointing because I was expecting to be able to release code that would work on the upcoming release,
>
What did we change/not include that caused this to not work?

 -Marius


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Auto Reload with lots of files causes massive slowdown again

nophead
If I load Mendel90 into the latest version I get 38000 warnings.

On Sun, 17 Feb 2019 at 07:46, nop head <[hidden email]> wrote:
The warnings about len and undef and the introduction of is_indef and is_list as work arounds.

On Sun, 17 Feb 2019 at 03:20, Marius Kintel <[hidden email]> wrote:
> On Feb 16, 2019, at 19:31, nop head <[hidden email]> wrote:
>
> The obvious solution would be git bisect but my code won't work on old versions any more because the language keeps changing. It won't even work on the release candidate, which is disappointing because I was expecting to be able to release code that would work on the upcoming release,
>
What did we change/not include that caused this to not work?

 -Marius


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Auto Reload with lots of files causes massive slowdown again

nophead
On the other hand it previews in about 5 minutes, whereas it used to take about 12 IIRC. Not many programs get faster!

On Sun, 17 Feb 2019 at 08:03, nop head <[hidden email]> wrote:
If I load Mendel90 into the latest version I get 38000 warnings.

On Sun, 17 Feb 2019 at 07:46, nop head <[hidden email]> wrote:
The warnings about len and undef and the introduction of is_indef and is_list as work arounds.

On Sun, 17 Feb 2019 at 03:20, Marius Kintel <[hidden email]> wrote:
> On Feb 16, 2019, at 19:31, nop head <[hidden email]> wrote:
>
> The obvious solution would be git bisect but my code won't work on old versions any more because the language keeps changing. It won't even work on the release candidate, which is disappointing because I was expecting to be able to release code that would work on the upcoming release,
>
What did we change/not include that caused this to not work?

 -Marius


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Auto Reload with lots of files causes massive slowdown again

nophead
It also works fine with Auto Reload so it actually seems to be something to do with my new machine design. I did change the way I include my library. Ironically instead of including every file everywhere I included just the files required. So I would expect it to be faster not many orders of magnitude slower. 

On thing that stands out is the paths in the cache are a mixture of forward / and \ because it depends whether the file is used via OPENSCADPATH or via a relative path. The StatCache doesn't have multiple entries, so I don't know why this causes a problem, but I have noticed a similar problem in the past. When I was inconsistent with the case of my library path it caused a big slowdown. I assumed that was because it doubled the size of Cache but maybe something else goes wrong.

For consistency I think all paths should use / inside OpenSCAD. I think \ is only needed on the command line in Windows, so I never use it C code. 

On Sun, 17 Feb 2019 at 08:05, nop head <[hidden email]> wrote:
On the other hand it previews in about 5 minutes, whereas it used to take about 12 IIRC. Not many programs get faster!

On Sun, 17 Feb 2019 at 08:03, nop head <[hidden email]> wrote:
If I load Mendel90 into the latest version I get 38000 warnings.

On Sun, 17 Feb 2019 at 07:46, nop head <[hidden email]> wrote:
The warnings about len and undef and the introduction of is_indef and is_list as work arounds.

On Sun, 17 Feb 2019 at 03:20, Marius Kintel <[hidden email]> wrote:
> On Feb 16, 2019, at 19:31, nop head <[hidden email]> wrote:
>
> The obvious solution would be git bisect but my code won't work on old versions any more because the language keeps changing. It won't even work on the release candidate, which is disappointing because I was expecting to be able to release code that would work on the upcoming release,
>
What did we change/not include that caused this to not work?

 -Marius


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Auto Reload with lots of files causes massive slowdown again

nophead

It is calling file system functions bypassing the cache. This will be tens of thousand of times as it is files squared with a library that is included everywhere.

On Sun, 17 Feb 2019 at 08:23, nop head <[hidden email]> wrote:
It also works fine with Auto Reload so it actually seems to be something to do with my new machine design. I did change the way I include my library. Ironically instead of including every file everywhere I included just the files required. So I would expect it to be faster not many orders of magnitude slower. 

On thing that stands out is the paths in the cache are a mixture of forward / and \ because it depends whether the file is used via OPENSCADPATH or via a relative path. The StatCache doesn't have multiple entries, so I don't know why this causes a problem, but I have noticed a similar problem in the past. When I was inconsistent with the case of my library path it caused a big slowdown. I assumed that was because it doubled the size of Cache but maybe something else goes wrong.

For consistency I think all paths should use / inside OpenSCAD. I think \ is only needed on the command line in Windows, so I never use it C code. 

On Sun, 17 Feb 2019 at 08:05, nop head <[hidden email]> wrote:
On the other hand it previews in about 5 minutes, whereas it used to take about 12 IIRC. Not many programs get faster!

On Sun, 17 Feb 2019 at 08:03, nop head <[hidden email]> wrote:
If I load Mendel90 into the latest version I get 38000 warnings.

On Sun, 17 Feb 2019 at 07:46, nop head <[hidden email]> wrote:
The warnings about len and undef and the introduction of is_indef and is_list as work arounds.

On Sun, 17 Feb 2019 at 03:20, Marius Kintel <[hidden email]> wrote:
> On Feb 16, 2019, at 19:31, nop head <[hidden email]> wrote:
>
> The obvious solution would be git bisect but my code won't work on old versions any more because the language keeps changing. It won't even work on the release candidate, which is disappointing because I was expecting to be able to release code that would work on the upcoming release,
>
What did we change/not include that caused this to not work?

 -Marius


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org