GNU logs - #53400, boring messages

Message sent to bug-coreutils@HIDDEN:

Subject: bug#53400: base64: decode Base64URL
Resent-From: Sergey Ponomarev <stokito@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Thu, 20 Jan 2022 21:33:01 +0000
Resent-Message-ID: <handler.53400.B.16427143593682 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 53400 <at> debbugs.gnu.org
MIME-Version: 1.0
From: Sergey Ponomarev <stokito@HIDDEN>
Date: Thu, 20 Jan 2022 23:31:55 +0200
Message-ID: <CADR0UcVx6EbHpehk1RxZu5gB8KCYXm7peK0B5GU24VATdrHzGA@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Package: coreutils

Currently the base64 util can't  parse base64 url safe.
RFC 3548 Base 64 Encoding with URL and Filename Safe Alphabet
https://datatracker.ietf.org/doc/html/rfc3548#page-6

But basically this is the same encoding: just two symbols differ and
no padding. So it's easy and safe just to parse both forms.
The coreutils have a separate command `basenc --base64url` to parse
the b64u but actually there is no any sense to make it as a separate
command.
Having this as a separate command makes it unusable for small Linux
distros like OpenWrt where the device's disk is very small.

Currently **many** shell scripts uses additional transformation like:

```sh
base64_padding()
{
  local len=$(( ${#1} % 4 ))
  local padded_b64=''
  if [ ${len} = 2 ]; then
    padded_b64="${1}=="
  elif [ ${len} = 3 ]; then
    padded_b64="${1}="
  else
    padded_b64="${1}"
  fi
  echo -n "$padded_b64"
}

base64url_to_b64()
{
  base64_padding "${1}" | tr -- '-_' '+/'
}
```

But this over complicates everything, reduces performance and is error prone.

I created a similar ticked for OpenSSL base64
https://github.com/openssl/openssl/issues/17559


Also as an additional request: can we add the new param `-D` in upper
case which will be an alias to `-d`.
The issue is that on MacOS the small -d was used for debug and they
used the -D in upper case. This creates a lot of problems for scripts
that should work on both Linux and MacOS. So most people use the long
--decode. But many users just don't know about this interop problem
and once I faced a problem when my college gave me a script that
didn't work for me.
This will prevent unexpected compatibility problems.

Regards,
Sergey Ponomarev

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Sergey Ponomarev <stokito@HIDDEN>
Subject: bug#53400: Acknowledgement (base64: decode Base64URL)
Message-ID: <handler.53400.B.16427143593682.ack <at> debbugs.gnu.org>
References: <CADR0UcVx6EbHpehk1RxZu5gB8KCYXm7peK0B5GU24VATdrHzGA@HIDDEN>
X-Gnu-PR-Message: ack 53400
X-Gnu-PR-Package: coreutils
Reply-To: 53400 <at> debbugs.gnu.org
Date: Thu, 20 Jan 2022 21:33:01 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-coreutils@HIDDEN

If you wish to submit further information on this problem, please
send it to 53400 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
53400: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D53400
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to bug-coreutils@HIDDEN:

Subject: bug#53400: base64: decode Base64URL
Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Fri, 21 Jan 2022 15:20:02 +0000
Resent-Message-ID: <handler.53400.B53400.16427783643592 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: Sergey Ponomarev <stokito@HIDDEN>, 53400 <at> debbugs.gnu.org
Message-ID: <3cc3ba54-d491-7a7c-ee5e-38251080dfcd@HIDDEN>
Date: Fri, 21 Jan 2022 15:18:13 +0000
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:95.0) Gecko/20100101
 Thunderbird/95.0
Content-Language: en-US
References: <CADR0UcVx6EbHpehk1RxZu5gB8KCYXm7peK0B5GU24VATdrHzGA@HIDDEN>
From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
In-Reply-To: <CADR0UcVx6EbHpehk1RxZu5gB8KCYXm7peK0B5GU24VATdrHzGA@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

On 20/01/2022 21:31, Sergey Ponomarev wrote:
> Package: coreutils
> 
> Currently the base64 util can't  parse base64 url safe.
> RFC 3548 Base 64 Encoding with URL and Filename Safe Alphabet
> https://datatracker.ietf.org/doc/html/rfc3548#page-6
> 
> But basically this is the same encoding: just two symbols differ and
> no padding. So it's easy and safe just to parse both forms.
> The coreutils have a separate command `basenc --base64url` to parse
> the b64u but actually there is no any sense to make it as a separate
> command.
> Having this as a separate command makes it unusable for small Linux
> distros like OpenWrt where the device's disk is very small.

Note coreutils can be built like busybox to include multiple commands in a single binary

> 
> Currently **many** shell scripts uses additional transformation like:
> 
> ```sh
> base64_padding()
> {
>    local len=$(( ${#1} % 4 ))
>    local padded_b64=''
>    if [ ${len} = 2 ]; then
>      padded_b64="${1}=="
>    elif [ ${len} = 3 ]; then
>      padded_b64="${1}="
>    else
>      padded_b64="${1}"
>    fi
>    echo -n "$padded_b64"
> }
> 
> base64url_to_b64()
> {
>    base64_padding "${1}" | tr -- '-_' '+/'
> }
> ```

I'm not sure about base64 accepting _either_ encoding without options.
There would be robustness issues with that.

Also blinding assuming padding may introduce robustness issues with truncated data.

It might be useful to have a "--decode-relaxed" option or something
to support decoding any b64 variant and assume no padding,
though preprocessing isn't that onerous, so this is marginal.

> But this over complicates everything, reduces performance and is error prone.
> 
> I created a similar ticked for OpenSSL base64
> https://github.com/openssl/openssl/issues/17559
> 
> 
> Also as an additional request: can we add the new param `-D` in upper
> case which will be an alias to `-d`.
> The issue is that on MacOS the small -d was used for debug and they
> used the -D in upper case. This creates a lot of problems for scripts
> that should work on both Linux and MacOS. So most people use the long
> --decode. But many users just don't know about this interop problem
> and once I faced a problem when my college gave me a script that
> didn't work for me.
> This will prevent unexpected compatibility problems.

macos using -D is unfortunate.
It's surprising they diverged from FreeBSD here.
The fact there is a cross platform --decode option,
suggests we don't need to add -D support,
as it doesn't help folks using -d on other systems.

cheers,
Pádraig

Message sent to bug-coreutils@HIDDEN:

Subject: bug#53400: base64: decode Base64URL
Resent-From: Sergey Ponomarev <stokito@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-coreutils@HIDDEN
Resent-Date: Fri, 21 Jan 2022 16:02:02 +0000
Resent-Message-ID: <handler.53400.B53400.16427809018043 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN>
Cc: 53400 <at> debbugs.gnu.org
MIME-Version: 1.0
References: <CADR0UcVx6EbHpehk1RxZu5gB8KCYXm7peK0B5GU24VATdrHzGA@HIDDEN>
 <3cc3ba54-d491-7a7c-ee5e-38251080dfcd@HIDDEN>
In-Reply-To: <3cc3ba54-d491-7a7c-ee5e-38251080dfcd@HIDDEN>
From: Sergey Ponomarev <stokito@HIDDEN>
Date: Fri, 21 Jan 2022 18:00:55 +0200
Message-ID: <CADR0UcWOgb6+4yy=vN1drHNt8YphBaHptZpUQFzDHNJnZvhAbA@HIDDEN>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

Thank you, P=C3=A1draig,

Your are absolutely right that for a better correctness it must be a
way to catch padding errors. And I just decided to check how it
behaves currently:

$ echo "Aladin" | base64
QWxhZGluCg=3D=3D

Now let's remove the padding:
$ echo "QWxhZGluCg" | base64 -d
Aladin
base64: invalid input
$ echo $?
1

So base64 decoded as much as it can but wrote a message into stderr
and returned an error code.
I think this is a good compromise: for those who are really want to
know about the padding they can check an error code.
We may add an additional error code to show that the base64 was
decoded as an Base64URL.

Also we may introduce an additional flag `-u` to say that we do expect
a Base64URL input and don't set the error code. The same flag may be
also used for base64 encoding.

Speaking about an additional flag the same idea was proposed in the
ticket for OpenSSL and I answered:

> Having it as a separate flag is a good thing to do for a better correctne=
ss. Still many unexpirienced users will fail just because they copied a fir=
st answer on SO. This makes the feature much less useful. Especially the bi=
g problem is that some base64 data may be decoded successfully during a dev=
elopment and testing and the code with bugs may be delivered to end users. =
Again, this is a very widely used tool.

So maybe the idea with decode both but set the exit code + additional
flag will be sufficient?

On Fri, 21 Jan 2022 at 17:19, P=C3=A1draig Brady <P@HIDDEN> wrote:
>
> On 20/01/2022 21:31, Sergey Ponomarev wrote:
> > Package: coreutils
> >
> > Currently the base64 util can't  parse base64 url safe.
> > RFC 3548 Base 64 Encoding with URL and Filename Safe Alphabet
> > https://datatracker.ietf.org/doc/html/rfc3548#page-6
> >
> > But basically this is the same encoding: just two symbols differ and
> > no padding. So it's easy and safe just to parse both forms.
> > The coreutils have a separate command `basenc --base64url` to parse
> > the b64u but actually there is no any sense to make it as a separate
> > command.
> > Having this as a separate command makes it unusable for small Linux
> > distros like OpenWrt where the device's disk is very small.
>
> Note coreutils can be built like busybox to include multiple commands in =
a single binary
>
> >
> > Currently **many** shell scripts uses additional transformation like:
> >
> > ```sh
> > base64_padding()
> > {
> >    local len=3D$(( ${#1} % 4 ))
> >    local padded_b64=3D''
> >    if [ ${len} =3D 2 ]; then
> >      padded_b64=3D"${1}=3D=3D"
> >    elif [ ${len} =3D 3 ]; then
> >      padded_b64=3D"${1}=3D"
> >    else
> >      padded_b64=3D"${1}"
> >    fi
> >    echo -n "$padded_b64"
> > }
> >
> > base64url_to_b64()
> > {
> >    base64_padding "${1}" | tr -- '-_' '+/'
> > }
> > ```
>
> I'm not sure about base64 accepting _either_ encoding without options.
> There would be robustness issues with that.
>
> Also blinding assuming padding may introduce robustness issues with trunc=
ated data.
>
> It might be useful to have a "--decode-relaxed" option or something
> to support decoding any b64 variant and assume no padding,
> though preprocessing isn't that onerous, so this is marginal.
>
> > But this over complicates everything, reduces performance and is error =
prone.
> >
> > I created a similar ticked for OpenSSL base64
> > https://github.com/openssl/openssl/issues/17559
> >
> >
> > Also as an additional request: can we add the new param `-D` in upper
> > case which will be an alias to `-d`.
> > The issue is that on MacOS the small -d was used for debug and they
> > used the -D in upper case. This creates a lot of problems for scripts
> > that should work on both Linux and MacOS. So most people use the long
> > --decode. But many users just don't know about this interop problem
> > and once I faced a problem when my college gave me a script that
> > didn't work for me.
> > This will prevent unexpected compatibility problems.
>
> macos using -D is unfortunate.
> It's surprising they diverged from FreeBSD here.
> The fact there is a cross platform --decode option,
> suggests we don't need to add -D support,
> as it doesn't help folks using -d on other systems.
>
> cheers,
> P=C3=A1draig



--=20
Sergey Ponomarev, skype:stokito

Message received at control <at> debbugs.gnu.org:

Received: (at control) by debbugs.gnu.org; 28 Jan 2022 01:58:04 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Jan 27 20:58:04 2022
Received: from localhost ([127.0.0.1]:57026 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1nDGWa-0005r8-8G
	for submit <at> debbugs.gnu.org; Thu, 27 Jan 2022 20:58:04 -0500
Received: from zimbra.cs.ucla.edu ([131.179.128.68]:59486)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <eggert@HIDDEN>) id 1nDGWY-0005qe-Pv
 for control <at> debbugs.gnu.org; Thu, 27 Jan 2022 20:58:03 -0500
Received: from localhost (localhost [127.0.0.1])
 by zimbra.cs.ucla.edu (Postfix) with ESMTP id 43C70160126
 for <control <at> debbugs.gnu.org>; Thu, 27 Jan 2022 17:57:57 -0800 (PST)
Received: from zimbra.cs.ucla.edu ([127.0.0.1])
 by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032)
 with ESMTP id 6G2tV34_eODM for <control <at> debbugs.gnu.org>;
 Thu, 27 Jan 2022 17:57:56 -0800 (PST)
Received: from localhost (localhost [127.0.0.1])
 by zimbra.cs.ucla.edu (Postfix) with ESMTP id A3920160133
 for <control <at> debbugs.gnu.org>; Thu, 27 Jan 2022 17:57:56 -0800 (PST)
X-Virus-Scanned: amavisd-new at zimbra.cs.ucla.edu
Received: from zimbra.cs.ucla.edu ([127.0.0.1])
 by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026)
 with ESMTP id ean83Se0DdVU for <control <at> debbugs.gnu.org>;
 Thu, 27 Jan 2022 17:57:56 -0800 (PST)
Received: from [192.168.1.9] (cpe-172-91-119-151.socal.res.rr.com
 [172.91.119.151])
 by zimbra.cs.ucla.edu (Postfix) with ESMTPSA id 80AA3160126
 for <control <at> debbugs.gnu.org>; Thu, 27 Jan 2022 17:57:56 -0800 (PST)
Message-ID: <9cd128fd-ef6d-2d10-db37-d4fc978ab51b@HIDDEN>
Date: Thu, 27 Jan 2022 17:57:56 -0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.5.0
Content-Language: en-US
To: control <at> debbugs.gnu.org
From: Paul Eggert <eggert@HIDDEN>
Subject: bug report maintenance
Organization: UCLA Computer Science Department
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: control
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

severity 53400 wishlist
severity 52525 wishlist
tags 42766 wontfix

Last modified: Fri, 28 Jan 2022 02:00:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.