Feature request: recognize the line break in multi line string

classic Classic list List threaded Threaded
34 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Feature request: recognize the line break in multi line string

runsun
The following code:
echo("abcdef1");  
echo("abc\ndef2");
echo("abc
def3");
generate:
ECHO: "abcdef1"
ECHO: "abc
def2"
ECHO: "abcdef3" // line break ignored
i.e, line breaks in a multi-line string is ignored. Is it possible to get openscad to recognize them ?
$ Runsun Pan, PhD
$ libs: scadx, doctest, faces(git), offline doc(git), runscad.py(2,git), editor of choice: CudaText ( OpenSCAD lexer); $ Tips; $ Snippets
tp3
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: recognize the line break in multi line string

tp3
On 09/13/2015 04:13 AM, runsun wrote:
> i.e, line breaks in a multi-line string is ignored. Is it possible to get
> openscad to recognize them ?
>
Possible yes, but I'm not sure it's a good idea. With the line
separator having at least 3 variations across platforms, that
could result in unexpected results depending on how the file
was saved.
Actually I'd rather consider that behavior for multi-line strings
a bug ;-). How are other scripting languages coping with that?
At least with bash, the string length differs when changing the
file format.

ciao,
  Torsten.



_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
-- Torsten
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: recognize the line break in multi line string

runsun
tp3 wrote
Actually I'd rather consider that behavior for multi-line strings
a bug ;-). How are other scripting languages coping with that?
At least with bash, the string length differs when changing the
file format.
In python we can do:
s='''abc
def'''
len(s)  ==> 7
s.split('\n') ==> ['abc', 'def']
$ Runsun Pan, PhD
$ libs: scadx, doctest, faces(git), offline doc(git), runscad.py(2,git), editor of choice: CudaText ( OpenSCAD lexer); $ Tips; $ Snippets
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: recognize the line break in multi line string

doug.moen
In reply to this post by tp3
The line separator has 2 variations: \n on Macintosh, Linux, BSD, and \r\n on Windows.

So change the lexer to ignore \r but preserve \n when scanning a string literal, and the results will be platform independent.

On 13 September 2015 at 06:36, Torsten Paul <[hidden email]> wrote:
On 09/13/2015 04:13 AM, runsun wrote:
> i.e, line breaks in a multi-line string is ignored. Is it possible to get
> openscad to recognize them ?
>
Possible yes, but I'm not sure it's a good idea. With the line
separator having at least 3 variations across platforms, that
could result in unexpected results depending on how the file
was saved.
Actually I'd rather consider that behavior for multi-line strings
a bug ;-). How are other scripting languages coping with that?
At least with bash, the string length differs when changing the
file format.

ciao,
  Torsten.



_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org




_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
tp3
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: recognize the line break in multi line string

tp3
On 09/13/2015 10:58 PM, doug moen wrote:
> The line separator has 2 variations: \n on Macintosh, Linux, BSD, and \r\n
> on Windows.
>
While newer MacOS does use \n, I would not be surprised to still
find the old \r separator in some cases.

Anyway, I guess the main question is if it should be changed or
not. The risk breaking existing scripts is probably not too
high.

Normalizing to always use \n makes sense and might even be
already done by the QTextStream used in GUI mode, not sure
about command line mode though.

ciao,
  Torsten.


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
-- Torsten
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: recognize the line break in multi line string

Alan Cox
In reply to this post by doug.moen
On Sun, 13 Sep 2015 16:58:00 -0400
doug moen <[hidden email]> wrote:

> The line separator has 2 variations: \n on Macintosh, Linux, BSD, and \r\n
> on Windows.
>
> So change the lexer to ignore \r but preserve \n when scanning a string
> literal, and the results will be platform independent.

There are systems that just use \r.

Just convert the files in format while moving them between systems.

Alan

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: recognize the line break in multi line string

doug.moen


Alan Cox said "There are systems that just use \r."

Not any more, and not for a long time. Apple dropped support for Mac OS 9 in 2002, and OpenSCAD seems to have started in 2009. Xt has also never supported MacOS 9. You've never been able to create a text file with \r line separators on any system that supports OpenSCAD, and those systems likely only exist in computer museums nowadays.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: recognize the line break in multi line string

G. Wade Johnson
On Sun, 13 Sep 2015 19:49:53 -0400
doug moen <[hidden email]> wrote:

> Alan Cox said "There are systems that just use \r."
>
> Not any more, and not for a long time. Apple dropped support for Mac
> OS 9 in 2002, and OpenSCAD seems to have started in 2009. Xt has also
> never supported MacOS 9. You've never been able to create a text file
> with \r line separators on any system that supports OpenSCAD, and
> those systems likely only exist in computer museums nowadays.

Less than a month ago, I got an example data file from a company that
was using \r as a line separator. I don't know for sure where they got
it. (Of course a quick one-liner converted it to something reasonable
for me.)

Until that time, I believed that you were correct. Now, I'm not sure I
would discount Alan's comment so quickly.

G. Wade
--
It's hard enough to find an error in your code when you're looking for
it; it's even harder when you've assumed your code is error-free.
                                                  -- Steve McConnell

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: recognize the line break in multi line string

doug.moen
Well, it turns out that \r line termination was also used by a number of popular 8 bit operating systems during the 1980s, including Apple II, Commodore Pet, TRS-80, and OS-9. And it seems that OS 9 continues to be used for embedded applications like in-car navigation systems and traffic light control systems.

But I don't know why we should care about supporting users who are editing their OpenSCAD source files on Apple IIs or embedded traffic light control computers.

On 13 September 2015 at 21:22, G. Wade Johnson <[hidden email]> wrote:
On Sun, 13 Sep 2015 19:49:53 -0400
doug moen <[hidden email]> wrote:

> Alan Cox said "There are systems that just use \r."
>
> Not any more, and not for a long time. Apple dropped support for Mac
> OS 9 in 2002, and OpenSCAD seems to have started in 2009. Xt has also
> never supported MacOS 9. You've never been able to create a text file
> with \r line separators on any system that supports OpenSCAD, and
> those systems likely only exist in computer museums nowadays.

Less than a month ago, I got an example data file from a company that
was using \r as a line separator. I don't know for sure where they got
it. (Of course a quick one-liner converted it to something reasonable
for me.)

Until that time, I believed that you were correct. Now, I'm not sure I
would discount Alan's comment so quickly.

G. Wade
--
It's hard enough to find an error in your code when you're looking for
it; it's even harder when you've assumed your code is error-free.
                                                  -- Steve McConnell

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org




_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: recognize the line break in multi line string

Alan Cox
In reply to this post by G. Wade Johnson
On Sun, 13 Sep 2015 20:22:19 -0500
"G. Wade Johnson" <[hidden email]> wrote:

> On Sun, 13 Sep 2015 19:49:53 -0400
> doug moen <[hidden email]> wrote:
>
> > Alan Cox said "There are systems that just use \r."
> >
> > Not any more, and not for a long time. Apple dropped support for Mac
> > OS 9 in 2002, and OpenSCAD seems to have started in 2009. Xt has also
> > never supported MacOS 9. You've never been able to create a text file
> > with \r line separators on any system that supports OpenSCAD, and
> > those systems likely only exist in computer museums nowadays.
>
> Less than a month ago, I got an example data file from a company that
> was using \r as a line separator. I don't know for sure where they got
> it. (Of course a quick one-liner converted it to something reasonable
> for me.)
>
> Until that time, I believed that you were correct. Now, I'm not sure I
> would discount Alan's comment so quickly.

Ignoring the \r isn't the way to do it. The logic is trivial to do it
right

  \r\n  -> one newline
else
  \r -> one newline
else
  \n -> one newline
else
  literal character

Alan


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: recognize the line break in multi line string

jdawgaz
I thought in most projects, the \n did the right thing DEPENDING UPON the platform it was running on.

So if you are on windows, \n becomes technically a \r\n, on mac and linux, just \n.

\r shouldn't even be in there, IMO.




--
Extra Ham Operator: K7AZJ
Registered Linux User: 275424
Raspberry Pi and Arduino developer

The most exciting phrase to hear in science - the one that heralds new discoveries - is not "Eureka!" but "That's funny...".
- Isaac. Asimov

If you give someone a program, you will frustrate them for a day; if you teach them how to program, you will frustrate them for a lifetime. 
- Anonymous

If writing good code requires very little comments, then writing really excellent code requires no comments at all!
- Ken Thompson


On Mon, Sep 14, 2015 at 3:13 AM, Alan Cox <[hidden email]> wrote:
On Sun, 13 Sep 2015 20:22:19 -0500
"G. Wade Johnson" <[hidden email]> wrote:

> On Sun, 13 Sep 2015 19:49:53 -0400
> doug moen <[hidden email]> wrote:
>
> > Alan Cox said "There are systems that just use \r."
> >
> > Not any more, and not for a long time. Apple dropped support for Mac
> > OS 9 in 2002, and OpenSCAD seems to have started in 2009. Xt has also
> > never supported MacOS 9. You've never been able to create a text file
> > with \r line separators on any system that supports OpenSCAD, and
> > those systems likely only exist in computer museums nowadays.
>
> Less than a month ago, I got an example data file from a company that
> was using \r as a line separator. I don't know for sure where they got
> it. (Of course a quick one-liner converted it to something reasonable
> for me.)
>
> Until that time, I believed that you were correct. Now, I'm not sure I
> would discount Alan's comment so quickly.

Ignoring the \r isn't the way to do it. The logic is trivial to do it
right

  \r\n  -> one newline
else
  \r -> one newline
else
  \n -> one newline
else
  literal character

Alan


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: recognize the line break in multi line string

runsun
In reply to this post by Alan Cox
Alan Cox wrote
  \r\n  -> one newline
else
  \r-> one newline
else
  \n -> one newline
else
  literal character
Maybe move \n to the 2nd condition ?
$ Runsun Pan, PhD
$ libs: scadx, doctest, faces(git), offline doc(git), runscad.py(2,git), editor of choice: CudaText ( OpenSCAD lexer); $ Tips; $ Snippets
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: recognize the line break in multi line string

Alan Cox
On Mon, 14 Sep 2015 07:10:21 -0700 (MST)
runsun <[hidden email]> wrote:

> Alan Cox wrote
> >   \r\n  -> one newline
> > else
> >   \r-> one newline
> > else
> >   \n -> one newline
> > else
> >   literal character
>
> Maybe move \n to the 2nd condition ?

Got a bikeshed to paint ?


_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: recognize the line break in multi line string

runsun
Alan Cox wrote
On Mon, 14 Sep 2015 07:10:21 -0700 (MST)
runsun <[hidden email]> wrote:

> Alan Cox wrote
> >   \r\n  -> one newline
> > else
> >   \r-> one newline
> > else
> >   \n -> one newline
> > else
> >   literal character
>
> Maybe move \n to the 2nd condition ?

Got a bikeshed to paint ?
The idea is to move more frequently used condition to the top so to reduce unnecessary checking.

But, maybe more checkings is better.
$ Runsun Pan, PhD
$ libs: scadx, doctest, faces(git), offline doc(git), runscad.py(2,git), editor of choice: CudaText ( OpenSCAD lexer); $ Tips; $ Snippets
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: recognize the line break in multi line string

MichaelAtOz
Administrator
Copying G.Wades's post from the other thread. My comment next.
G. Wade Johnson wrote
On Mon, 14 Sep 2015 06:21:33 -0700
Jerry Davis <[hidden email]> wrote:

> I thought in most projects, the \n did the right thing DEPENDING UPON
> the platform it was running on.
>
> So if you are on windows, \n becomes technically a \r\n, on mac and
> linux, just \n.
>
> \r shouldn't even be in there, IMO.

Whichever tool you are working with normally handles line endings
correctly for the platform they are on (modulo little issues like
Windows text/binary modes).

The problem comes when you get files from a different system...

Some tools can handle the variations, and we are talking about adding
that feature to OpenSCAD. It isn't really as much of a problem in the
whitespace, but when a newline occur in a string, more thought is
needed.

Just my 0.02.
G. Wade
Admin - email* me if you need anything,
or if I've done something stupid...
* click on my MichaelAtOz label, there is a link to email me.

Unless specifically shown otherwise above, my contribution is in the Public Domain; to the extent possible under law, I have waived all copyright and related or neighbouring rights to this work.
Obviously inclusion of works of previous authors is not included in the above.


The TPP is no simple “trade agreement.” Fight it! http://www.ourfairdeal.org/ time is running out!
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: recognize the line break in multi line string

MichaelAtOz
Administrator
In reply to this post by Alan Cox
Alan Cox wrote
Got a bikeshed to paint ?
You should really include a reference when you post such.
Admin - email* me if you need anything,
or if I've done something stupid...
* click on my MichaelAtOz label, there is a link to email me.

Unless specifically shown otherwise above, my contribution is in the Public Domain; to the extent possible under law, I have waived all copyright and related or neighbouring rights to this work.
Obviously inclusion of works of previous authors is not included in the above.


The TPP is no simple “trade agreement.” Fight it! http://www.ourfairdeal.org/ time is running out!
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: recognize the line break in multi line string

MichaelAtOz
Administrator
In reply to this post by MichaelAtOz
What we are talking about syntactically is Line continuation.

The OP wishes the CR character to be a continuation (not a token) when within the scope of a text string, ie between two string tokens ("), as default behaviour.

While I have not seen examples of multi-line strings in the wild, I can see how it would be used with the current behaviour.

string="
a bunch of lines,
a bunch of lines,
a bunch of lines,
\n
a bunch of lines,
a bunch of lines,
a bunch of lines,
\n
a bunch of lines,
a bunch of lines,
a bunch of lines,
\r
";

As such I think the chances of breaking existing code is high.

I think that the OP issue is, probably, more of a problem with command-line use. Perhaps a run-time flag?
--stringseperator:[none|preserve|platform|LF|CRLF]

But in summary, I think it is much easier for the operating system/scripts etc to fix it, than to try to interpret what the intent is.

Finally, given what the following does, I don't think it should be changed.

s1="def\nghi";
s2="jkl\r\nmno";
s3="jkl
\n                                      // comment - actually string text
nmno";

s=s3;
for(i=[0:len(s)-1]) {
        echo(s[i]);
}
Admin - email* me if you need anything,
or if I've done something stupid...
* click on my MichaelAtOz label, there is a link to email me.

Unless specifically shown otherwise above, my contribution is in the Public Domain; to the extent possible under law, I have waived all copyright and related or neighbouring rights to this work.
Obviously inclusion of works of previous authors is not included in the above.


The TPP is no simple “trade agreement.” Fight it! http://www.ourfairdeal.org/ time is running out!
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: recognize the line break in multi line string

runsun
In reply to this post by MichaelAtOz
G. Wade Johnson wrote
The problem comes when you get files from a different system...

Some tools can handle the variations, and we are talking about adding
that feature to OpenSCAD. It isn't really as much of a problem in the
whitespace, but when a newline occur in a string, more thought is
needed.
Python has much larger user space and a lot more fields of application. If python can handle files from different systems, then this cross-platform issue shouldn't become a problem for openscad.


$ Runsun Pan, PhD
$ libs: scadx, doctest, faces(git), offline doc(git), runscad.py(2,git), editor of choice: CudaText ( OpenSCAD lexer); $ Tips; $ Snippets
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: recognize the line break in multi line string

bobc
In reply to this post by MichaelAtOz
MichaelAtOz wrote
The OP wishes the CR character to be a continuation (not a token) when within the scope of a text string, ie between two string tokens ("), as default behaviour.
Really? I thought he was asking for the opposite.

There does seem to be a lot of talking at cross purposes here. What we are talking about here is specifically how are control characters handled in quoted strings.

OpenSCAD uses a lexer built with flex. The code for strings looks like this:

\" { BEGIN(cond_string); stringcontents.clear(); }
<cond_string>{
\\n { stringcontents += '\n'; }
\\t { stringcontents += '\t'; }
\\r { stringcontents += '\r'; }
\\\\ { stringcontents += '\\'; }
\\\" { stringcontents += '"'; }
{UNICODE}               { parser_error_pos -= strlen(lexertext) - 1; stringcontents += lexertext; }
\\x[0-7]{H}             { unsigned long i = strtoul(lexertext + 2, NULL, 16); stringcontents += (i == 0 ? ' ' : (unsigned char)(i & 0xff)); }
\\u{H}{4}|\\U{H}{6}     { char buf[8]; to_utf8(lexertext + 2, buf); stringcontents += buf; }
[^\\\n\"] { stringcontents += lexertext; }
\" { BEGIN(INITIAL);
                        parserlval.text = strdup(stringcontents.c_str());
                        return TOK_STRING; }
}

The sequences "\n" and "\r" are de-escaped and added to the string. I can't figure out where raw CR and LF are handled, but they appear to be ignored?
tp3
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: recognize the line break in multi line string

tp3
Von: bobc <[hidden email]>
> The sequences "\n" and "\r" are de-escaped and added to the string. I can't
> figure out where raw CR and LF are handled, but they appear to be ignored?
>
I did not fully test all the combinations yet, but I think the unquoted \n
is explicitely _not_ handled in

[^\\\n\"] { stringcontents += lexertext; }

hence it's ignored. As I did not get any stray \r in GUI mode when using
DOS file format, so I guess the QTextStream already translates everything
to just \n. Command line might be different as it's not using Qt.

As said before, the main question is not how to implement it, but IF it
should be changed (or just put on the big heap that is wishes for the
OpenSCAD2 language ;-).

ciao,
  Torsten.

_______________________________________________
OpenSCAD mailing list
[hidden email]
http://lists.openscad.org/mailman/listinfo/discuss_lists.openscad.org
-- Torsten
12