Remove Whole-Word IRL Language Translations from DMFI Language Tool

Suggestions Should Be Posted in Their Respective Categories

Moderators: Moderator, Quality Control, Developer, DM

Post Reply
User avatar
RoseFrost
Posts: 30
Joined: Wed Mar 01, 2023 9:12 pm

Remove Whole-Word IRL Language Translations from DMFI Language Tool

Unread post by RoseFrost »

Preface:
The DMFI language tool mostly operates as a substitution cipher. Each letter that gets typed into the tool is replaced by one or multiple letters/characters. This creates an effective illusion of a foreign language—something which players cannot read. It also gives a good sense of the general character of the language by how the cipher is set up. Is a language vowel heavy and flowing or guttural and clipped, etc.

In addition to the substitution cipher, there is also a look-up table which replaces some whole words in their entirety. Here is an example in Elven. 'Hello' gets translated to 'Alae', while the individual letters would actually make up 'Iracce' via the cipher. This is well and fine, as 'Alae' is the correct lore translation.
Image

Problem:
The problem begins, when that look-up table translates worlds into IRL languages. The biggest offender that I've found is Illuskan, but I'm sure there are others. Here is an example:
Image

This is obviously just German, which is a problem on three levels:

1. The language tool is using a simple look-up table. There is no remote attempt at grammar. No conjugation, declension, adjustment of word order, etc, etc, etc. It is like nails on a chalkboard for anyone who actually speaks the language being subjected to this barbarity.

2. Even given the butchered grammar, speakers of the IRL language in question can often get an unfair OOC/meta advantage in being able to interpret the untranslated language. While the above example is a grammatical mess, it's still clear what is meant. Even Google Translate can figure it out.

3. It is a violation of established lore. There simply is no one-to-one equivalent between Forgotten Realms and IRL languages. Amn and Cormyr—the Spain and France of Faerun—both speak Chondathan (Chondathan doesn't appear to use a look-up table to either language, and is just a cipher). While Illuskan is described as being inspired by 'North Germanic', the three actual lore translations we have are far more Norse. Dock-Alfar for dark elves comes from the Old Norse dökkálfar. Hamfriggan and Eigerstor are the remaining two words given in lore. Only 'Eiger' sounds more German than Scandinavian.

Solution:
Simply remove all instances where a look-up table translates whole words from an FR language to an IRL language. Every language has a substitution cipher at its core, and if these words are simply pulled from their respective look-up tables, the cipher will take over and the problem will be solved.

For languages like Elven, Drow, and Alzhedo where we do have an extensive lore dictionary, whole-word translation is appropriate and good.

In terms of actually implementing the solution, I imagine that would just take going through some text file and deleting things.
User avatar
Aspect of Sorrow
Custom Content
Posts: 2633
Joined: Fri Mar 28, 2014 7:11 pm
Location: Reliquary

Re: Remove Whole-Word IRL Language Translations from DMFI Language Tool

Unread post by Aspect of Sorrow »

Given it's not Tolkien level of specifics, we could anagram RL detected words to mostly meet the theme without it being the RL word.
User avatar
gedweyignasia
Custom Content
Posts: 1353
Joined: Sat Nov 23, 2013 11:27 pm
Location: EST/UTC-4
Contact:

Re: Remove Whole-Word IRL Language Translations from DMFI Language Tool

Unread post by gedweyignasia »

RoseFrost wrote: Wed Jul 23, 2025 12:46 pm The DMFI language tool mostly operates as a substitution cipher. Each letter that gets typed into the tool is replaced by one or multiple letters/characters. This creates an effective illusion of a foreign language—something which players cannot read. It also gives a good sense of the general character of the language by how the cipher is set up. Is a language vowel heavy and flowing or guttural and clipped, etc.

In addition to the substitution cipher, there is also a look-up table which replaces some whole words in their entirety. Here is an example in Elven. 'Hello' gets translated to 'Alae', while the individual letters would actually make up 'Iracce' via the cipher. This is well and fine, as 'Alae' is the correct lore translation.
This is correct. NWScript is a limited language, but someone managed to do a pretty good job here in spite of that. (Working both within the constraints of being computationally efficient and lacking sophisticated data structures.) Throwing in some RP-specific words to satisfy players who've learned some of FR's languages was a nice touch.
RoseFrost wrote: Wed Jul 23, 2025 12:46 pm 1. The language tool is using a simple look-up table. There is no remote attempt at grammar. No conjugation, declension, adjustment of word order, etc, etc, etc. It is like nails on a chalkboard for anyone who actually speaks the language being subjected to this barbarity.
That's disappointing that RL languages would be included in the lookup table instead of just FR languages. As you correctly identified, this "translation" is just a substitution cipher, and is not up to the task of proper grammar. In RL applications, that requires complex transformer models, which often still perform poorly.
RoseFrost wrote: Wed Jul 23, 2025 12:46 pm 2. Even given the butchered grammar, speakers of the IRL language in question can often get an unfair OOC/meta advantage in being able to interpret the untranslated language. While the above example is a grammatical mess, it's still clear what is meant. Even Google Translate can figure it out.
The same is true for players who learn words of FR's languages, or who recognize the substitution cipher and can parse the original message. It's also true for players who are told OOCly what the message said. The server has rules against metagaming, and they apply in all situations.
RoseFrost wrote: Wed Jul 23, 2025 12:46 pm Simply remove all instances where a look-up table translates whole words from an FR language to an IRL language. Every language has a substitution cipher at its core, and if these words are simply pulled from their respective look-up tables, the cipher will take over and the problem will be solved.
This is a reasonable approach, but given that it's only intermittent words and these languages are intended to resemble (and were inspired by) real-world languages, it might be better to address the inaccuracies, if someone is interested in undertaking this project.
RoseFrost wrote: Wed Jul 23, 2025 12:46 pm In terms of actually implementing the solution, I imagine that would just take going through some text file and deleting things.
Unfortunately, that's not the case. Our substitution ciphers are basically a bunch of nested if-then statements. (Ideal would be a state machine, but I didn't write it. In either case, it's not just hitting a CSV with word pairs on every line.)
User avatar
DaloLorn
Posts: 2466
Joined: Tue Mar 26, 2019 2:44 am
Location: Discord (@dalolorn)

Re: Remove Whole-Word IRL Language Translations from DMFI Language Tool

Unread post by DaloLorn »

gedweyignasia wrote: Wed Jul 23, 2025 8:04 pm
RoseFrost wrote: Wed Jul 23, 2025 12:46 pm In terms of actually implementing the solution, I imagine that would just take going through some text file and deleting things.
Unfortunately, that's not the case. Our substitution ciphers are basically a bunch of nested if-then statements. (Ideal would be a state machine, but I didn't write it. In either case, it's not just hitting a CSV with word pairs on every line.)
... It... literally is, though. It's not a CSV, per se, but the ciphers and word lists are sourced from 2DAs, with an input and output string for every letter of the alphabet, followed by similar entries for as many words as the author bothered to translate. And you know enough about how 2DAs look to realize that this results in a word pair on every line.

For instance, Illuskan is defined in bg_soz_tools/csl_lang_illuskan.2da.

Mind you, your description of what the code does with the contents of those 2DAs is accurate enough to my recollection. Just missed the point where you don't need to touch any actual code here.
European player, UTC+1 (+2 during DST). Ex-fixer of random bits. Active in Discord.
Active characters:
  • Zeila Linepret
  • Ilhara Evrine
  • Linathyl Selmiyeritar
  • Belinda Ravenblood
  • Virin Swifteye
  • Gurzhuk
User avatar
gedweyignasia
Custom Content
Posts: 1353
Joined: Sat Nov 23, 2013 11:27 pm
Location: EST/UTC-4
Contact:

Re: Remove Whole-Word IRL Language Translations from DMFI Language Tool

Unread post by gedweyignasia »

DaloLorn wrote: Thu Jul 24, 2025 3:26 am
gedweyignasia wrote: Wed Jul 23, 2025 8:04 pm
RoseFrost wrote: Wed Jul 23, 2025 12:46 pm In terms of actually implementing the solution, I imagine that would just take going through some text file and deleting things.
Unfortunately, that's not the case. Our substitution ciphers are basically a bunch of nested if-then statements. (Ideal would be a state machine, but I didn't write it. In either case, it's not just hitting a CSV with word pairs on every line.)
... It... literally is, though. It's not a CSV, per se, but the ciphers and word lists are sourced from 2DAs, with an input and output string for every letter of the alphabet, followed by similar entries for as many words as the author bothered to translate. And you know enough about how 2DAs look to realize that this results in a word pair on every line.

For instance, Illuskan is defined in bg_soz_tools/csl_lang_illuskan.2da.

Mind you, your description of what the code does with the contents of those 2DAs is accurate enough to my recollection. Just missed the point where you don't need to touch any actual code here.
Oh! That's very convenient! I thought it was all hardcoded in branch statements.
User avatar
DaloLorn
Posts: 2466
Joined: Tue Mar 26, 2019 2:44 am
Location: Discord (@dalolorn)

Re: Remove Whole-Word IRL Language Translations from DMFI Language Tool

Unread post by DaloLorn »

There's some hardcoding going on, but IIRC it's for special characters used to modify the system's behavior (e.g. @ for a verbatim block, * for an emote, or ^ to force a single-character translation), or characters that should be treated as equivalent to whitespace or perhaps even nothing while trying to determine word boundaries.

There's also special rules for nonverbal languages like Drowsign or Animal Empathy, where ciphertext would be inappropriate and a semi-random emote is used instead. But those, like the rest of the NWScript involved, are beyond the scope of the OP's request and can be safely ignored. And of course there's the language mapper itself, figuring out which 2DA belongs to a given language.
European player, UTC+1 (+2 during DST). Ex-fixer of random bits. Active in Discord.
Active characters:
  • Zeila Linepret
  • Ilhara Evrine
  • Linathyl Selmiyeritar
  • Belinda Ravenblood
  • Virin Swifteye
  • Gurzhuk
User avatar
Zar'shalee
Posts: 62
Joined: Tue May 12, 2015 4:04 am

Re: Remove Whole-Word IRL Language Translations from DMFI Language Tool

Unread post by Zar'shalee »

I personally do not even use language tool that often. I find it clunky a rough around the edges. It would help greatly if we could seamlessly choose languages in normal chat window by simple slash commands, for example /und "text" (undercommon) /dr "text" (drow), etc. Instead of writing into a different, dedicated window. Would that be possible to implement?
Zar'shalee Tor'viir - Matron mother of house Tor'viir, Sol d'Lolth
User avatar
Zeikari
Posts: 92
Joined: Fri Feb 03, 2023 9:02 am

Re: Remove Whole-Word IRL Language Translations from DMFI Language Tool

Unread post by Zeikari »

Zar'shalee wrote: Tue Jul 29, 2025 5:28 am I personally do not even use language tool that often. I find it clunky a rough around the edges. It would help greatly if we could seamlessly choose languages in normal chat window by simple slash commands, for example /und "text" (undercommon) /dr "text" (drow), etc. Instead of writing into a different, dedicated window. Would that be possible to implement?
You can sort of do it already! If you open the language tool, you can use it to just select a language, then just click away, and chat in the regular chat normally as you would for Common. So long as the language tool is open and has another language selected, stuff written in the regular chat will be auto-translated!

Aside from being smoother, you also have the advantage of the three speaking dots appearing on your character's head, which prevents people from talking over you.
User avatar
DaloLorn
Posts: 2466
Joined: Tue Mar 26, 2019 2:44 am
Location: Discord (@dalolorn)

Re: Remove Whole-Word IRL Language Translations from DMFI Language Tool

Unread post by DaloLorn »

Yeah, the dedicated window is only mandatory if you're trying to perform acts of ventriloquism on your minions.
European player, UTC+1 (+2 during DST). Ex-fixer of random bits. Active in Discord.
Active characters:
  • Zeila Linepret
  • Ilhara Evrine
  • Linathyl Selmiyeritar
  • Belinda Ravenblood
  • Virin Swifteye
  • Gurzhuk
User avatar
Zar'shalee
Posts: 62
Joined: Tue May 12, 2015 4:04 am

Re: Remove Whole-Word IRL Language Translations from DMFI Language Tool

Unread post by Zar'shalee »

Zeikari wrote: Tue Jul 29, 2025 6:37 am
Zar'shalee wrote: Tue Jul 29, 2025 5:28 am I personally do not even use language tool that often. I find it clunky a rough around the edges. It would help greatly if we could seamlessly choose languages in normal chat window by simple slash commands, for example /und "text" (undercommon) /dr "text" (drow), etc. Instead of writing into a different, dedicated window. Would that be possible to implement?
You can sort of do it already! If you open the language tool, you can use it to just select a language, then just click away, and chat in the regular chat normally as you would for Common. So long as the language tool is open and has another language selected, stuff written in the regular chat will be auto-translated!

Aside from being smoother, you also have the advantage of the three speaking dots appearing on your character's head, which prevents people from talking over you.
If that is true, dont tell me it wouldn't be possible to make it more seamless and smooth without an extra chat window popping on...
Zar'shalee Tor'viir - Matron mother of house Tor'viir, Sol d'Lolth
User avatar
Zeikari
Posts: 92
Joined: Fri Feb 03, 2023 9:02 am

Re: Remove Whole-Word IRL Language Translations from DMFI Language Tool

Unread post by Zeikari »

I wouldn't know about what could or could not be changed to make it smoother, but it's definitely possible!
I always use the main dialogue chat when I speak in other languages, and I frequently talk in non-Common languages on most my pcs.
User avatar
DaloLorn
Posts: 2466
Joined: Tue Mar 26, 2019 2:44 am
Location: Discord (@dalolorn)

Re: Remove Whole-Word IRL Language Translations from DMFI Language Tool

Unread post by DaloLorn »

Zar'shalee wrote: Tue Jul 29, 2025 8:14 am
Zeikari wrote: Tue Jul 29, 2025 6:37 am
Zar'shalee wrote: Tue Jul 29, 2025 5:28 am I personally do not even use language tool that often. I find it clunky a rough around the edges. It would help greatly if we could seamlessly choose languages in normal chat window by simple slash commands, for example /und "text" (undercommon) /dr "text" (drow), etc. Instead of writing into a different, dedicated window. Would that be possible to implement?
You can sort of do it already! If you open the language tool, you can use it to just select a language, then just click away, and chat in the regular chat normally as you would for Common. So long as the language tool is open and has another language selected, stuff written in the regular chat will be auto-translated!

Aside from being smoother, you also have the advantage of the three speaking dots appearing on your character's head, which prevents people from talking over you.
If that is true, dont tell me it wouldn't be possible to make it more seamless and smooth without an extra chat window popping on...
Yes and no.

Yes: The system technically only cares about the value of a localvar that is set in any context where the language chat is opened.

No:
  • You'd need a whole extra mapping of languages to their chat commands (which would be > commands, by the way, e.g. ">und", because we can't add slash commands serverside).
  • There is no currently accepted way of changing that localvar without closing (and possibly reopening) the language chat, and if such a mechanism were added, the language chat would need to be made aware of it to avoid undefined behavior where the user changed their language while the language chat was open.
European player, UTC+1 (+2 during DST). Ex-fixer of random bits. Active in Discord.
Active characters:
  • Zeila Linepret
  • Ilhara Evrine
  • Linathyl Selmiyeritar
  • Belinda Ravenblood
  • Virin Swifteye
  • Gurzhuk
Post Reply

Return to “Suggestions and Discussion”