GNU bug report logs - #55112
[Issue 8 drafts 0001556]: clarify meaning of \n used in a bracket expression in a sed context address or s-command

Previous Next

To reply to this bug, email your comments to 55112 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Christoph Anton Mitterer <calestyo <at> scientia.org>
Cc: Geoff Clare <gwc <at> opengroup.org>, bug-sed <at> gnu.org,
 austin-group-l <at> opengroup.org
Subject: Re: [Issue 8 drafts 0001556]: clarify meaning of \n used in a
 bracket expression in a sed context address or s-command
Date: Mon, 25 Apr 2022 11:06:23 -0500

Adding bug-sed <at> gnu.org into this conversation.

On Mon, Apr 25, 2022 at 02:50:22AM +0200, Christoph Anton Mitterer via austin-group-l at The Open Group wrote:
> Hey.
> 
> Geoff, I haven't had time yet to look at your updated proposal of
> #1550, not sure whether I manage to do it this night or in the next
> days.
> But I'll definitely reply, so please be a bit more patient. :-)
> 
> 
> However, on thing came to my minds again, which I think needs further
> discussion...
> 
> 
> 
> The current "solution" to a number of previous problems is:
> 
> Inside a bracket expression there cannot be any escape sequences.
> Therefore, there cannot be any \n (in the sense of <newline>) nor any
> \c (in the sense of "un-delimitering" the delimiter character c).
> 
> 
> While this is per se perfectly valid (and solves numerous issues), it
> has one problem:
> 
> (at least) GNU sed breaks it already!
> 
> 
> 
> As you noted yourself in
> https://www.austingroupbugs.net/view.php?id=1556#c5621
> 
> it requires POSIXLY_CORRECT=1 to work as it should.
> 
> $ printf 'a\\b\n' | sed 's/a[\n]b/X/'
> a\b
> $ printf 'a\nb\n' | sed 's/a[\n]b/X/'
> a
> b
> $ printf 'a\nb\n' | sed -z 's/a[\n]b/X/'
> X
> $ printf 'anb\n' | sed 's/a[\n]b/X/'
> anb
> $ export POSIXLY_CORRECT=1
> $ printf 'a\\b\n' | sed 's/a[\n]b/X/'
> X
> $ printf 'a\nb\n' | sed 's/a[\n]b/X/'
> a
> b
> $ printf 'a\nb\n' | sed -z 's/a[\n]b/X/'
> a
> b
> $ printf 'anb\n' | sed 's/a[\n]b/X/'
> X
> $ 
> 
> 
> NOT so for GNU's extension of '\s':
> '\s'
>      Matches whitespace characters (spaces and tabs).  Newlines
>      embedded in the pattern/hold spaces will also match...
> (and I assume neither for any similar such extensions):
> 
> $ printf 'asb\n' | sed 's/a[\s]b/X/'
> X
> $ printf 'a\\b\n' | sed 's/a[\s]b/X/'
> X
> $ printf 'a b\n' | sed 's/a[\s]b/X/'
> a b
> $ export POSIXLY_CORRECT=1
> $ printf 'asb\n' | sed 's/a[\s]b/X/'
> X
> calestyo <at> heisenberg:~$ printf 'a\\b\n' | sed 's/a[\s]b/X/'
> X
> calestyo <at> heisenberg:~$ printf 'a b\n' | sed 's/a[\s]b/X/'
> a b
> $
> 
> 
> It also works as expected for escaped delimiter characters:
> $ printf 'aDb\n' | sed 'sDa[\D]bDXD'
> X
> $ printf 'a\\b\n' | sed 'sDa[\D]bDXD'
> X
> 
> even when the delimiter char has also special meaning when escaped (as
> with '\s'):
> $ printf 'asb\n' | sed 'ssa[\s]bsXs'
> X
> $ printf 'a\\b\n' | sed 'ssa[\s]bsXs'
> X
> $ printf 'a b\n' | sed 'ssa[\s]bsXs'
> a b
> 
> 
> (all the above with GNU sed 4.8).
> 
> 
> So the only problematic case seems to be '\n'.
> 
> 
> 
> I don't want to step on anyone's toes... but GNU sed is probably one of
> the (if not the) major implementation of sed, isn't it?
> 
> 
> And regardless of POSIXLY_CORRECT, the standard describes now a
> behaviour (namely that the bracket expression [\n] is the literal
> characters '\' or 'n' and *not* <newline>)... which is not shared by a
> major implementation, at least not with its default settings.
> 
> Anyone who reads the standard would assume that [\n] is not a
> <newline>. 
> And of course we could just say "well your implementation is not
> compliant" or "look at it's documentation, where it says about
> POSIXLY_CORRECT" ... but that doesn't seem so good to me.
> 
> Usually, implementations extend POSIX rather gracefully, but this is a
> more serious deviation.
> 
> 
> I mean should we just leave it at that?
> 
> Or should we add some hint, e.g. indicating that portable applications
> should not use '\n' but rather 'n\' ... or perhaps even generally place
> '\' last in the bracket expression?
> 
> 
> The best would of course be to get GNU change it's behaviour, though I
> have no idea how likely that is ;-)
> 
> I had tried to reach out to GNU and BusyBox sed maintainers before, and
> while I got replies from BusyBox' I couldn't get in touch with GNU's.
> 
> Is there anyone who's in contact with these people?

The GNU sed developers can be reached at bug-sed <at> gnu.org (per the
output of 'sed --help', and as done in this email).

So if I'm restating your complaint correctly, you are worried that GNU
sed's non-POSIX behavior (what you get by default when POSIXLY_CORRECT
is not set) treats the four-byte sequence '[\n]' in an s-command regex
as a bracket expression for the single character of a literal newline
(that is, interpreting \n as an escape sequence even though it is
inside a bracket expression), instead of as a bracket expression for
either of a literal backslash or literal n; but concur that its
behavior when being POSIX-compliant matches the POSIX rules.

POSIX can't control what GNU sed does when in non-POSIX mode.  But it
can document a recommendation to spell the bracket expression intended
to match either a backslash or an n in the order [n\] to avoid any
potential confusion with [\n] being interpreted as an escape sequence.

Or am I missing something else that you are proposing that either the
Austin Group should do in its documentation efforts, and/or which GNU
sed should do to comply with the recent Austin Group recommendations?

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Message #8 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Christoph Anton Mitterer <calestyo <at> scientia.org>
To: Eric Blake <eblake <at> redhat.com>
Cc: Geoff Clare <gwc <at> opengroup.org>, bug-sed <at> gnu.org,
 austin-group-l <at> opengroup.org
Subject: Re: [Issue 8 drafts 0001556]: clarify meaning of \n used in a
 bracket expression in a sed context address or s-command
Date: Tue, 26 Apr 2022 01:44:21 +0200

Hey Eric.

On Mon, 2022-04-25 at 11:06 -0500, Eric Blake wrote:
> The GNU sed developers can be reached at bug-sed <at> gnu.org (per the
> output of 'sed --help', and as done in this email).

Ah, I think I had written to sed-devel in January.


> So if I'm restating your complaint correctly

"complaint" is a bit harsh ;-) ... it's not my intention to step on
anyone's toes, just to hoping to help with portability.


> you are worried that GNU
> sed's non-POSIX behavior (what you get by default when
> POSIXLY_CORRECT
> is not set)

Speaking of POSIXLY_CORRECT ... I'm not sure how much that really helps
in practise.

First, the reality probably is that most users won't read the info page
from top to bottom and even if they do, it's not for sure that they
really understand the implications of e.g. '[\n]' and that they'd need
to use POSIXLY_CORRECT.
Sure you can argue now that this is then the fault of the user, but I
don't think that this helps in practise.

Second, (sed) scripts may flow in both directions, i.e. from an
implementation that is (per default) POSIXly correct to GNU sed (which
per default is not) - and vice versa.
So when such script comes from a non-GNU-sed and uses '[\n]' in the
strict POSIX sense, it would likely be just used as is with GNU sed,
that it has different semantics is possibly not immaculately visible,
as there's no error or so, and thus people probably won't realise that
they'd need to set POSIXLY_CORRECT non empty for such "foreign"
scripts.
An the same would likely happen in the other direction. The average GNU
sed user may perhaps never notice that '[\n]' being newline is a GNU
speciality unless he knows the standard well. If that is then used on a
non-GNU-sed, sematics change again.


> treats the four-byte sequence '[\n]' in an s-command regex
> as a bracket expression for the single character of a literal newline
> (that is, interpreting \n as an escape sequence even though it is
> inside a bracket expression), instead of as a bracket expression for
> either of a literal backslash or literal n; but concur that its
> behavior when being POSIX-compliant matches the POSIX rules.

I guess it's at least quite unfortunate that it does so.
Especially because GNU seems to really do this only with sed, e.g. grep
(with POSIXLY_CORRECT UNset) seems to interpret '[\n]' POSIXly
correct...

$ printf 'a\nb' | grep -z '^a[\n]b$' ; echo

$ printf 'a\\b' | grep -z '^a[\n]b$' ; echo
a\b
$ printf 'anb' | grep -z '^a[\n]b$' ; echo
anb

... which I'd blindly guess is also not necessarily clear to the
average GNU grep/sed user.


> POSIX can't control what GNU sed does when in non-POSIX mode.

Sure... and even if it would do so in POSIX mode, there's no POSIX
police ;-)

Nevertheless... in practise most people will just assume that the
default mode is mostly POSIX compliant, except perhaps for "graceful"
extensions.

All these GNU extensions (like '\+' and friends for BREs... or '\s' and
friends for BREs and EREs) still work nicely with POSIX, cause POSIX
says that these produce undefined results, so if someone really wanted
to be portable, he didn't use it.

But this is different for the sed + '[\n]' case. Some who restricted
himself to just POSIX would still get into troubles.

And sure, strictly speaking you're of course right, and only with
POSIXLY_CORRECT non-empty, GNU sed is guaranteed to behave so - but
again, I'd blindly guess that in practise that goes quite easily
unnoticed.


> But it
> can document a recommendation to spell the bracket expression
> intended
> to match either a backslash or an n in the order [n\] to avoid any
> potential confusion with [\n] being interpreted as an escape
> sequence.

The problem remains of course for any scripts which are written&tested
with sed implementations that behave the other way and which are then
used with GNU sed.

The best (for portability) would probably if GNU sed could change the
behaviour, but I see of course that unfortunately this is likely not
easily possible either.

I just searched the sed info page... and that seems to basically say:
> '[LIST]'
> '[^LIST]'
>     Matches any single character in LIST: for example, '[aeiou]'
>     matches all vowels.  A list may include sequences like
>     'CHAR1-CHAR2', which matches any character between (inclusive)
>     CHAR1 and CHAR2.  *Note Character Classes and Bracket
>     Expressions::.
...
a bit further down
...
> '\n'
>      Matches the newline character.

IMO, that's however "outside" of the part for bracket expressions,
because everything else that is described on the same level (like '\+'
or '\DIGIT') is clearly *not* intended to work inside GNU sed bracket
expression, right?

However later in "5.5 Character Classes and Bracket Expressions":
> Also, when not in 'POSIXLY_CORRECT' mode, special escapes like '\n'
> and '\t' are recognized within LIST.  *Note Escapes::.

So I guess at this point it's game over and GNU sed could never really
change behaviour without breaking gazillion things.


btw: I'd hope that these \<char> escape sequences produce at least all
the literal <char>, when <char> is also the delimiter.



> Or am I missing something else that you are proposing that either the
> Austin Group should do in its documentation efforts, and/or which GNU
> sed should do to comply with the recent Austin Group recommendations?

Well I guess given that GNU sed explicitly documented this behaviour
for the non-'POSIXLY_CORRECT'-mode) means that there cannot anything be
done than documenting it as good as possible (on both sides).

Perhaps better to use '\\' for any literally meant <backslash>, than to
just put it at the end of the list, cause some implementations could
also think about giving special meaning to '\]'.


Really unfortunate though, especially that it's then not even
consistent across GNU (i.e. also in GNU sed).


Thanks,
Chris.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.