test_pretty_print.py no longer works on the MSYS2 build on Win7

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

test_pretty_print.py no longer works on the MSYS2 build on Win7

nophead
ctest fails over like this:

$ ctest -R nonascii
running 'C:/msys64/home/ChrisP/openscad/tests/../Release/openscad.exe --info' to generate sysinfo.txt
Post test:C:/msys64/mingw64/bin/python.exe "C:/msys64/home/ChrisP/openscad/tests/test_pretty_print.py" --builddir=C:/msys64/home/ChrisP/openscad/tests
Test project C:/msys64/home/ChrisP/openscad/tests
Enforcing Default test configuration. Use ctest -C <config> to override
    Start 1105: openscad-nonascii_sfære
1/1 Test #1105: openscad-nonascii_sfære .........***Failed    0.17 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.31 sec

The following tests FAILED:
        1105 - openscad-nonascii_sfære (Failed)
Traceback (most recent call last):
  File "C:/msys64/home/ChrisP/openscad/tests/test_pretty_print.py", line 532, in <module>
    main()
  File "C:/msys64/home/ChrisP/openscad/tests/test_pretty_print.py", line 497, in main
    startdate, tests, enddate, imgcomparer = parselog(testlog)
  File "C:/msys64/home/ChrisP/openscad/tests/test_pretty_print.py", line 187, in parselog
    text = data.decode('utf-8')
  File "C:/msys64/mingw64/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe6 in position 1071: invalid continuation byte
Problem running command: C:/msys64/mingw64/bin/python.exe "C:/msys64/home/ChrisP/openscad/tests/test_pretty_print.py" --builddir=C:/msys64/home/ChrisP/openscad/tests
Problem executing post-test command(s).
Errors while running CTest


This test has always failed for me but this change now makes the test system fail. I assume that the console output from OpenSCAD is not utf-8 encoded in this environment but I don't really know what the correct solution is.


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: test_pretty_print.py no longer works on the MSYS2 build on Win7

nophead
The main reason test_pretty_print.py falls over seems to be that the log it receives has a mixture of latin1 and utf8 encoding so nothing can convert it to unicode properly.

The log comes from test_cmdline_tool.py and that also falls over in python2.7 due to this line: https://github.com/openscad/openscad/blob/master/tests/test_cmdline_tool.py#L245 which adds an ascii string to a unicode string. Under Python3 both strings are unicode so it will work.

So it looks like since I last used it mods have been made to make it Python3 compatible. Is it still supposed to work on Python2?

One of the problems is sys.argv[] are ascii strings with latin encoding in Python2 on Win7. Presumably they are already unicode in Python3 so I think it would need code condition on the version just to handle the args. And on Linux Python7 are they utf8?

Should I chase down the Python2 bugs or try to upgrade MSYS2 to Python3? Having recently upgraded my native Python to 3 this is a whole world of pain, manly because they changed the string type to unicode and you can no longer use 8 bit ascii without encoding it.



On 23 August 2018 at 17:31, nop head <[hidden email]> wrote:
ctest fails over like this:

$ ctest -R nonascii
running 'C:/msys64/home/ChrisP/openscad/tests/../Release/openscad.exe --info' to generate sysinfo.txt
Post test:C:/msys64/mingw64/bin/python.exe "C:/msys64/home/ChrisP/openscad/tests/test_pretty_print.py" --builddir=C:/msys64/home/ChrisP/openscad/tests
Test project C:/msys64/home/ChrisP/openscad/tests
Enforcing Default test configuration. Use ctest -C <config> to override
    Start 1105: openscad-nonascii_sfære
1/1 Test #1105: openscad-nonascii_sfære .........***Failed    0.17 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.31 sec

The following tests FAILED:
        1105 - openscad-nonascii_sfære (Failed)
Traceback (most recent call last):
  File "C:/msys64/home/ChrisP/openscad/tests/test_pretty_print.py", line 532, in <module>
    main()
  File "C:/msys64/home/ChrisP/openscad/tests/test_pretty_print.py", line 497, in main
    startdate, tests, enddate, imgcomparer = parselog(testlog)
  File "C:/msys64/home/ChrisP/openscad/tests/test_pretty_print.py", line 187, in parselog
    text = data.decode('utf-8')
  File "C:/msys64/mingw64/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe6 in position 1071: invalid continuation byte
Problem running command: C:/msys64/mingw64/bin/python.exe "C:/msys64/home/ChrisP/openscad/tests/test_pretty_print.py" --builddir=C:/msys64/home/ChrisP/openscad/tests
Problem executing post-test command(s).
Errors while running CTest


This test has always failed for me but this change now makes the test system fail. I assume that the console output from OpenSCAD is not utf-8 encoded in this environment but I don't really know what the correct solution is.



_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
tp3
Reply | Threaded
Open this post in threaded view
|

Re: test_pretty_print.py no longer works on the MSYS2 build on Win7

tp3
On 08/25/2018 01:06 PM, nop head wrote:
> So it looks like since I last used it mods have been made to
> make it Python3 compatible. Is it still supposed to work on
> Python2?
>
I don't know if all the build environments that are in use
will support Python3. So Python2 support might still be
needed at this point.

I guess going forward by getting rid of Python2 is the
better option. Making the MSYS2 env work with Python3 would
be a step into that direction.

We don't have any automated builds for MSYS2, so issues
tend to be detected late. Are there any CI providers that
would allow running MSYS2 builds?

ciao,
   Torsten.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
-- Torsten
Reply | Threaded
Open this post in threaded view
|

Re: test_pretty_print.py no longer works on the MSYS2 build on Win7

nophead
So I played around with this for a few days and came to the conclusion it isn't practical to make the same code work on Python2 and Python3 once unicode with non ascii characters is involved.

The way to do it in Python2 is with 8 bit str strings. The program then doesn't care about encoding, it just handles 8 bits transparently. The OS handles which characters the 8 bit codes represent. That all works fine with Western languages as long we all sing from the same codepage. It doesn't work well if you want more than one language at the same time or more than 256 glyphs.

Python3 strings are unicode and it seems to automatically promote the args from the OS encoding and be able to print unicode, automatically doing the output encoding to match the output device.

In Python2 the args come in in latin1 and need to be presented to the subprocess in latin1. This presumable is platform and language dependent. You can reliably print unicode without encoding it to bytes but if you do that in Python3 it prints the b" prefix. So the only way to make a program work on both is with lots of conditional code, which is ugly and not easy to maintain.

I took the plunge and updated my MSYS2 to Python3 with pacman -S python3. The only issue with that is it installs it to /usr/bin but Python2 is in /mingw64/bin and that is higher in the search order. I had to manually nuke Python2, I don't know if there is a better way to do it.

After upgrading I found some bugs in test_pretty_print.py that I don't think are anything to do with my platform. I have submitted a PR to fix that: https://github.com/openscad/openscad/pull/2467.






On 25 August 2018 at 13:55, Torsten Paul <[hidden email]> wrote:
On 08/25/2018 01:06 PM, nop head wrote:
So it looks like since I last used it mods have been made to
make it Python3 compatible. Is it still supposed to work on
Python2?

I don't know if all the build environments that are in use
will support Python3. So Python2 support might still be
needed at this point.

I guess going forward by getting rid of Python2 is the
better option. Making the MSYS2 env work with Python3 would
be a step into that direction.

We don't have any automated builds for MSYS2, so issues
tend to be detected late. Are there any CI providers that
would allow running MSYS2 builds?

ciao,
  Torsten.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: test_pretty_print.py no longer works on the MSYS2 build on Win7

nophead
4th paragraph should say "You can't reliably print unicode without encoding it to bytes"

On 29 August 2018 at 11:58, nop head <[hidden email]> wrote:
So I played around with this for a few days and came to the conclusion it isn't practical to make the same code work on Python2 and Python3 once unicode with non ascii characters is involved.

The way to do it in Python2 is with 8 bit str strings. The program then doesn't care about encoding, it just handles 8 bits transparently. The OS handles which characters the 8 bit codes represent. That all works fine with Western languages as long we all sing from the same codepage. It doesn't work well if you want more than one language at the same time or more than 256 glyphs.

Python3 strings are unicode and it seems to automatically promote the args from the OS encoding and be able to print unicode, automatically doing the output encoding to match the output device.

In Python2 the args come in in latin1 and need to be presented to the subprocess in latin1. This presumable is platform and language dependent. You can reliably print unicode without encoding it to bytes but if you do that in Python3 it prints the b" prefix. So the only way to make a program work on both is with lots of conditional code, which is ugly and not easy to maintain.

I took the plunge and updated my MSYS2 to Python3 with pacman -S python3. The only issue with that is it installs it to /usr/bin but Python2 is in /mingw64/bin and that is higher in the search order. I had to manually nuke Python2, I don't know if there is a better way to do it.

After upgrading I found some bugs in test_pretty_print.py that I don't think are anything to do with my platform. I have submitted a PR to fix that: https://github.com/openscad/openscad/pull/2467.






On 25 August 2018 at 13:55, Torsten Paul <[hidden email]> wrote:
On 08/25/2018 01:06 PM, nop head wrote:
So it looks like since I last used it mods have been made to
make it Python3 compatible. Is it still supposed to work on
Python2?

I don't know if all the build environments that are in use
will support Python3. So Python2 support might still be
needed at this point.

I guess going forward by getting rid of Python2 is the
better option. Making the MSYS2 env work with Python3 would
be a step into that direction.

We don't have any automated builds for MSYS2, so issues
tend to be detected late. Are there any CI providers that
would allow running MSYS2 builds?

ciao,
  Torsten.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org



_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org