126863 – en_AU.dic has UTF-8 errors

Issue 126863 - en_AU.dic has UTF-8 errors

Summary: en_AU.dic has UTF-8 errors

Status:	CLOSED FIXED

Alias:	None

Product:	General
Classification:	Code
Component:	spell checking (show other issues)
Version:	4.2.0-dev
Hardware:	All All

Importance:	P5 (lowest) Normal (vote)
Target Milestone:	---
Assignee:	AOO issues mailing list
QA Contact:

URL:
Keywords:

Depends on:
Blocks:

Reported:	2016-03-09 12:57 UTC by Tom Anderson
Modified:	2023-08-18 14:40 UTC (History)
CC List:	5 users (show)

See Also:
Issue Type:	DEFECT
Latest Confirmation in:	4.1.2
Developer Difficulty:	---

Attachments
en_AU - accents fixes (240.38 KB, patch) 2016-03-14 10:13 UTC, marcoagpinto	no flags	Details \| Diff
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this issue.

Description Tom Anderson 2016-03-09 12:57:02 UTC

In regards to the en_AU.dic extension for Australian spelling, a number of spellings were corrupted. This appears to have occurred due to incorrect conversion to/from UTF-8 during adding new words or in the editing process in 2008, but these errors persist to the current version of the en_AU.dic. I would fix these errors myself but surely there is a maintainer to contact in regards to this issue? Has it occurred with other dictionaries?

Two options I see, delete all entries with characters that are not Australian English, or change all those bad characters to good ones. Noting that the character ï¿½ implies error, not a particular character. In other words we see variants such as piï¿½ata (should be piñata) and clichï¿½ (should be cliché).

I tracked this down through various versions of the en_AU.dic http://extensions.services.openoffice.org/en/project/AustralianDictionary

Here is some analysis of version and line numbers of 2 words as they changed over time. This problem is rife in the newest version of en_AU.dic, with at least 211 occurrences of the ¿ character, which indicates a failed conversion. The word cliche, for example, is misrepresented over time in different ways. Note that many words with the ï¿½ character in the en_AU.dic file never appeared correctly, although this example for the word cliché was originally correct but was corrupted over time.

Version 2016.03.01 (Newest)
1700: clichï¿½/SM

Version: 2010.03.16
1700: clichï¿½/SM

Version: 2008.11.25
1700: clich�/SM

Version: 2008.10.3
1702: cliché/MS
1703: clichï¿½/SM

Version: 1.0.0
1523: cliché/MS

With reference to files at:
http://extensions.services.openoffice.org/en/project/english-dictionaries-apache-openoffice
http://extensions.services.openoffice.org/en/project/AustralianDictionary

Comment 1 Marcus 2016-03-09 19:39:36 UTC

@Marco:
Please can you have a look? Maybe you can explain and solve the issue. Thanks.

Comment 2 marcoagpinto 2016-03-09 22:33:48 UTC

I have opened the .AFF + .DIC and I confirm that there are corrupted words.

The person who converted the .DIC to UTF-8 probably did something wrong in the procedure.

I am the maintainer of the English dictionaries but I only add words to the British one.

The other dictionaries are only packed by me in the monthly OXT.

@Tom, is there any chance you could fix the words since you know what to search for?

If you do it, please also update the .AFF and README with your name and state that you fixed the issue.

Then, ZIP and upload the files here and in the monthly update it will be fixed for everyone.

Thanks!

:-)

Comment 3 Andrea Pescetti 2016-03-13 09:34:22 UTC

Words to be fixed are about 200. The following search lists all "suspicious" entries, each with its line number in the en_AU.dic file. It is likely that some smart find/replace can fix the file easily.

$ grep -n -v -e "^[a-zA-Z0-9/'-\. \!]*$" en_AU.dic 
1104:bourï¿½e
1110:boutonniï¿½re/SM
1626:chï¿½telaine/SM
1700:clichï¿½/SM
1701:clichï¿½d
1713:cloisonnï¿½/M
1882:Concepciï¿½n/M
2083:coulï¿½e/SM
2202:crï¿½pe/SM
2468:derriï¿½re/S
2533:diamantï¿½
2604:discothï¿½que/SM
2647:dï¿½modï¿½
2721:dï¿½paysï¿½e
2761:Dï¿½sseldorf
2762:d'ï¿½tre
3010:entrecï¿½te/SM
3011:entrepï¿½t/S
3786:glacï¿½/DGS
4919:kindergï¿½rtner/SM
5209:littï¿½rateur/S
5331:macramï¿½/MS
5458:matinï¿½e/S
5461:maï¿½tre
5652:ï¿½migrï¿½/S
5730:mï¿½nage
5985:nï¿½e
6047:nï¿½gligï¿½
6196:Nounï¿½a
6759:piï¿½on/S
7123:purï¿½eing
7480:rï¿½gime/SM
7540:rï¿½le/MS
8002:shouldâve
8190:Sï¿½o
8208:soignï¿½
8209:soirï¿½e/SM
8731:Tannhï¿½user/M
9119:tï¿½te-bï¿½che
9120:tï¿½te-ï¿½-tï¿½te
10250:appliquï¿½/SMG
10251:appliquï¿½d
10381:attachï¿½/MS
10808:Bogotï¿½/M
11388:chï¿½teau/MS
11445:ï¿½clat/M
11660:confrï¿½re/MS
11819:cortï¿½ge/SM
11896:crï¿½che/MS
11940:cruditï¿½s
12044:Danaï¿½
12075:dï¿½colletage/S
12076:dï¿½coupage
12391:divorcï¿½
12392:divorcï¿½e/SM
12472:dï¿½paysï¿½
12477:dï¿½railleur/SM
12505:dï¿½tente/S
12924:exposï¿½/SM
12954:faï¿½ade/MS
13301:Franï¿½oise/M
13687:Grï¿½newald/M
13702:Gï¿½teborg/M
14541:jalapeï¿½o/S
15118:lycï¿½e
15228:manquï¿½/M
15294:matï¿½riel/MS
15527:mï¿½lï¿½e/SM
15528:mï¿½moire
15663:mï¿½tier/S
15784:naï¿½vety/S
16016:Noumï¿½a/M
16370:passï¿½/M
16824:premiï¿½re/DMGS
16834:pï¿½res/F
16995:purï¿½e/DSM
17101:raison d'ï¿½tre
17465:rï¿½sumï¿½/S
17491:Rï¿½union/M
17801:seï¿½orita/SM
18913:touchï¿½
19393:voilï¿½
19772:abbï¿½/S
19872:adiï¿½s
20700:blasï¿½
21017:cafï¿½/SM
21838:crï¿½pey
21978:dï¿½but/S
21980:dï¿½colletï¿½
22316:dï¿½nouement
22492:Dvorï¿½k/M
22929:Fabergï¿½/M
23037:fiancï¿½/MS
23275:Franï¿½ois
23589:grï¿½ce
23932:Hï¿½loise/M
24445:jardiniï¿½re/MS
24494:Josï¿½/M
24703:lamï¿½
24717:ï¿½lan/M
24952:Lomï¿½/M
25096:Mallarmï¿½/M
25383:mightâve
25709:naivetï¿½/MS
25745:naï¿½vetï¿½/S
26162:outrï¿½
27135:recherchï¿½
27421:rosï¿½
27443:rouï¿½/SM
27541:sï¿½ance/MS
27776:seï¿½ora/SM
27777:Seï¿½ora/M
28142:soupï¿½on/SM
28900:Tomï¿½/M
29414:vis-ï¿½-vis
30136:animï¿½
30291:arï¿½te/MS
30341:Asunciï¿½n/M
30549:Bartï¿½k/M
30579:bï¿½che
30854:bouclï¿½
30935:bric-ï¿½-brac
31646:compï¿½re/M
31723:consommï¿½/S
32070:dï¿½classï¿½
32071:dï¿½classï¿½e
32073:dï¿½cor/MS
32369:dï¿½jï¿½
32406:doppelgï¿½nger
32633:Elysï¿½e/M
32762:Esterhï¿½zy/M
32920:faï¿½ence/S
33018:fiancï¿½e/MS
33269:frappï¿½
33422:Gï¿½del/M
33484:Gewï¿½rztraminer
33731:habituï¿½/SM
34305:ingï¿½nue/S
35087:Lumiï¿½re/M
35494:Mï¿½nchhausen/M
35719:naï¿½veness
36676:portiï¿½re/SM
36721:prï¿½cis/dMS
36869:protï¿½gï¿½/SM
36870:protï¿½gï¿½e/S
36903:pï¿½tï¿½/M
37244:repï¿½chage
37552:sautï¿½/GSD
37596:Schrï¿½dinger/M
37754:seï¿½ores
38976:Tï¿½rshavn/M
39263:Velï¿½squez/M
39308:vicuï¿½a/S
39889:aide-mï¿½moire
40658:ï»¿blowhole
40864:bï¿½te/S
40865:bï¿½tise
41015:canapï¿½/S
41322:ï¿½clair/MS
41360:clientï¿½le/M
41490:communiquï¿½/SM
41741:coupï¿½/SM
41840:croï¿½ton/SM
41853:Cï¿½te
41960:dï¿½bï¿½cle/MS
41962:dï¿½butante/MS
41963:dï¿½colletï¿½e
42395:dï¿½shabillï¿½'s
42678:entrï¿½e/S
43040:fï¿½hn
43041:fï¿½hrer/SM
43108:flambï¿½/DSG
43315:fï¿½te/SM
43385:ï¿½gar
43397:garï¿½on/MS
43668:Gruyï¿½re
44086:howâd
45140:maï¿½ana/M
45216:manï¿½ge/GDS
45346:Mï¿½bius
45604:mï¿½lange
45803:nï¿½
45854:naï¿½ve/Y
45881:negligï¿½e/SM
46200:olï¿½
46501:passï¿½e
46688:piï¿½ata/S
47063:Provenï¿½al
47489:risquï¿½
47872:seï¿½or/M
48226:soufflï¿½/SM
49086:tï¿½te
49797:Yaoundï¿½/M

Comment 4 marcoagpinto 2016-03-14 10:13:13 UTC

Created attachment 85357 [details]
en_AU - accents fixes

Here is the fixed .DIC .

I auto-replaced all corrupted characters with an "é" and then had to check the entire .DIC because over 90% of é's were other characters with accents.

It seems that the corrupted symbol was all the same.

Please tell me if you find any invalid words with accents.

Comment 5 Matthias Seidel 2023-08-07 15:31:53 UTC

Are these fixes in the latest en_AU dictionary?

If yes, can we close this issue?

Comment 6 Matthias Seidel 2023-08-17 10:39:58 UTC

I assume this is fixed, at least I couldn't find these errors in the latest en_AU dictionary...

If you disagree, feel free to reopen.

Comment 7 Tom Anderson 2023-08-18 00:46:57 UTC

(In reply to Matthias Seidel from comment #6)
> I assume this is fixed, at least I couldn't find these errors in the latest
> en_AU dictionary...
> 
> If you disagree, feel free to reopen.

I also checked and the errors are not there. All the words with the corrupted accents are now without any accents, e.g. like "cliché/MS" is now "cliche/MDS". Should work just fine.

Comment 8 Matthias Seidel 2023-08-18 14:40:54 UTC

Hi,

thank you  for the confirmation.

I want to close some old issues here. ;-)