GNU bug report logs - #31318
emacs-26 fns-tests-collate-sort failure on glibc 2.27

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 31318 in the body.
You can then email your comments to 31318 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: bug-gnu-emacs <at> gnu.org
Subject: emacs-26 fns-tests-collate-sort failure on glibc 2.27
Date: Mon, 30 Apr 2018 10:36:31 -0700

[Message part 1 (text/plain, inline)]

I upgraded to Fedora 28 beta over the weekend (Fedora 28 is scheduled 
for release next week) and ran into one test case failure on the 
emacs-26 branch: fns-tests-collate-sort fails (test/src/fns-tests.log is 
attached). Looking into the test, it appears that the test is making 
unportable assumptions about strcoll behavior in non-POSIX locales, an 
assumption that is not true of glibc 2.27 as shipped in Fedora 28. 
Proposed patch attached.

[fns-tests.log (text/x-log, attachment)]

[0001-Port-collation-tests-to-glibc-2.27.txt (text/plain, attachment)]

Message #10 received at 31318 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 31318 <at> debbugs.gnu.org
Subject: Re: bug#31318: emacs-26 fns-tests-collate-sort failure on glibc 2.27
Date: Mon, 30 Apr 2018 22:50:29 +0300

> From: Paul Eggert <eggert <at> cs.ucla.edu>
> Date: Mon, 30 Apr 2018 10:36:31 -0700
> 
> I upgraded to Fedora 28 beta over the weekend (Fedora 28 is scheduled 
> for release next week) and ran into one test case failure on the 
> emacs-26 branch: fns-tests-collate-sort fails (test/src/fns-tests.log is 
> attached). Looking into the test, it appears that the test is making 
> unportable assumptions about strcoll behavior in non-POSIX locales, an 
> assumption that is not true of glibc 2.27 as shipped in Fedora 28. 

Really?  So you are saying that 'ls' now sorts differently in UTF-8
locales on Fedora 28?  That is, it no longer ignores punctuation in
file names?  Doesn't that cause user outcry?

> -  ;; In a language specific locale, collation order is different.
> -  (should (string-collate-lessp
> -	   "xyzzy" "XYZZY"
> -	   (if (eq system-type 'windows-nt) "enu_USA" "en_US.UTF-8")))
> +  ;; In a language specific locale on MS-Windows, collation order is different.
> +  (when (eq system-type 'windows-nt)
> +    (should (string-collate-lessp "xyzzy" "XYZZY" "en_US.UTF-8")))

This is backwards: Windows needs enu_USA, not en_US.UTF-8 (which is
not supported on Windows).

> -  ;; Diacritics are sorted between similar letters for other locales.
> -  (should
> -   (equal
> -    (sort '("Ævar" "Agustín" "Adrian" "Eli")
> -	  (lambda (a b)
> -	    (let ((w32-collate-ignore-punctuation t))
> -	      (string-collate-lessp
> -	       a b (if (eq system-type 'windows-nt) "enu_USA" "en_US.UTF-8")))))
> -    '("Adrian" "Ævar" "Agustín" "Eli"))))
> +  ;; Diacritics are sorted between similar letters for other locales,
> +  ;; on MS-Windows systems.
> +  (when (eq system-type 'windows-nt)
> +    (should
> +     (equal
> +      (sort '("Ævar" "Agustín" "Adrian" "Eli")
> +            (lambda (a b)
> +              (let ((w32-collate-ignore-punctuation t))
> +                (string-collate-lessp
> +                 a b "enu_USA"))))
> +      '("Adrian" "Ævar" "Agustín" "Eli")))))

So none of this now works on Fedora 28?  That's too bad, don't you
think?  Why did they change the collation order?

Message #13 received at 31318 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 31318 <at> debbugs.gnu.org
Subject: Re: bug#31318: emacs-26 fns-tests-collate-sort failure on glibc 2.27
Date: Mon, 30 Apr 2018 17:39:57 -0700

[Message part 1 (text/plain, inline)]

On 04/30/2018 12:50 PM, Eli Zaretskii wrote:
>
> you are saying that 'ls' now sorts differently in UTF-8
> locales on Fedora 28?  That is, it no longer ignores punctuation in
> file names?  Doesn't that cause user outcry?

Yes, that's what I'm saying, at least for en_US.utf8. I haven't heard 
any user outcry. Not that I've been listening for one....

> This is backwards: Windows needs enu_USA, not en_US.UTF-8 (which is
> not supported on Windows).

Thanks, that was a silly typo. Revised patch attached, for the emacs-26 
branch.

> Why did they change the collation order?


To be more consistent with other platforms, as I understand it. The idea 
is to do collation consistent with ISO/IEC 14651 
<https://www.iso.org/standard/68309.html>. For more details, please see 
<https://sourceware.org/bugzilla/show_bug.cgi?id=14095>.

Regardless of whether one thinks the change was a good idea, the Emacs 
tests should be portable to any POSIX platform, and since POSIX does not 
specify how the en_US.utf8 locale works the tests should not insist on 
either the old glibc implementation or the new one.

[0001-Port-collation-tests-to-glibc-2.27.patch (text/x-patch, attachment)]

Message #16 received at 31318 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 31318 <at> debbugs.gnu.org
Subject: Re: bug#31318: emacs-26 fns-tests-collate-sort failure on glibc 2.27
Date: Tue, 01 May 2018 19:11:02 +0300

> Cc: 31318 <at> debbugs.gnu.org
> From: Paul Eggert <eggert <at> cs.ucla.edu>
> Date: Mon, 30 Apr 2018 17:39:57 -0700
> 
> Regardless of whether one thinks the change was a good idea, the Emacs 
> tests should be portable to any POSIX platform, and since POSIX does not 
> specify how the en_US.utf8 locale works the tests should not insist on 
> either the old glibc implementation or the new one.

Absolutely.  But IMO it'd be a pity not to be able to test this
important functionality on Posix platforms.  So how about replacing
these tests with something that works in a non-en_US locale in a way
that is different from en_US?  The functions being tested signal an
error if a locale passed to them is not installed, so it should be
possible to test whether a locale is available before running a test
with it.

WDYT?

Message #19 received at 31318 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 31318 <at> debbugs.gnu.org
Subject: Re: bug#31318: emacs-26 fns-tests-collate-sort failure on glibc 2.27
Date: Tue, 1 May 2018 09:31:30 -0700

On 05/01/2018 09:11 AM, Eli Zaretskii wrote:
> how about replacing
> these tests with something that works in a non-en_US locale in a way
> that is different from en_US?

Something along those lines could be done (it's done in coreutils with 
French), but it's a bit tricky so presumably it should be done in 
master. In contrast the patch I'm proposing is designed for emacs-26.

Message #22 received at 31318 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 31318 <at> debbugs.gnu.org
Subject: Re: bug#31318: emacs-26 fns-tests-collate-sort failure on glibc 2.27
Date: Tue, 01 May 2018 19:49:22 +0300

> Cc: 31318 <at> debbugs.gnu.org
> From: Paul Eggert <eggert <at> cs.ucla.edu>
> Date: Tue, 1 May 2018 09:31:30 -0700
> 
> On 05/01/2018 09:11 AM, Eli Zaretskii wrote:
> > how about replacing
> > these tests with something that works in a non-en_US locale in a way
> > that is different from en_US?
> 
> Something along those lines could be done (it's done in coreutils with 
> French), but it's a bit tricky so presumably it should be done in 
> master. In contrast the patch I'm proposing is designed for emacs-26.

I'm okay with applying your patch to emacs-26 and then re-adding tests
on master.

Thanks.

Message #27 received at 31318-done <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: 31318-done <at> debbugs.gnu.org
Subject: Re: emacs-26 fns-tests-collate-sort failure on glibc 2.27
Date: Tue, 18 Jun 2019 16:52:58 -0700

This patch was installed into the emacs-26 branch last year but I forgot 
to close the bug report. Closing it now.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.