I’m no stranger to citing blog posts and web pages. Several useful and prominent resources are sometimes just hosted on a random webpage, or some service like HuggingFace or Kaggle. No shame in citing DataCanary et al. when you need to. In this case, however, the Kaggle linkhttps://www.kaggle.com/competitions/quora-question-pairs isn’t actually a dataset; it’s a competition, as you might’ve guessed from the URL. Predictably, it’s by Quora and is about classifying duplicate pairs:

Currently, Quora uses a Random Forest model to identify duplicate questions. In this competition, Kagglers are challenged to tackle this natural language processing problem by applying advanced techniques to classify whether question pairs are duplicates or not.

Competitions come with their own datasets, and the licensing for that is set by each competition runner. Good news for this one: the license is clearly visible and provided for the data, as part of the competition rules. Here’s a key excerpt:

…Participants must use the Data solely for the purpose and duration of the Competition…

Oops. Quora Question Pairs turns out to not be licensed for research use! Except not, because while the Kaggle link is the first thing that comes up when you search it, the third one down is the dataset on the Quora website itself. Surprise! It has actual authors!

Specifically, Shankar Iyer, Nikhil Dandekar, and Kornél Csernai. So why exactly is renowned author hilfialkaff DataCanary cited instead?

Just to be clear, the Kaggle one isn’t a unique permutation of it or anything: it’s the exact same dataset.The Kaggle competition includes a test set, but I checked the data they used myself for any contamination, and none of the unique questions in the test set were sampled. This isn’t just one paper, either. LLM2Vec took their dataset concoction from Springer et al, who took it from Wang et al., both of whom attribute QQP to hilfialkaff DataCanary. In fact, a rudimentary search brought up twenty-three other papers (and one thesis statement!) that cite good old DataCanary when they should be citing Iyer.

Outside of being plain embarrassing, I think this is indicative of a pervasive attitude in NLP and AI research in general: a lack of care for dataset licensing and proper attribution. This is a particularly egregious example, and I’m genuinely surprised no one across these thirty-odd papers caught it (hilfialkaff DataCanary? Really?), but there are probably several similar misattributions that are sneakier in a lot of papers. As much as hunting for licenses is a pain in the ass, I think it’s a fairly good thing to expect papers to have a thorough licenses section (mine does it!). ACL seems to be moving towards that with their Responsible NLP checklist,https://aclrollingreview.org/responsibleNLPresearch but I don’t think it’s particularly enforced in any way. Might be good to get on that. Regardless, just check your citations, people.

A different, language-specific set of transformations would be needed to equivalently normalize non-English text, but due to our lack of linguistic knowledge to build such normalizers for all languages, we resort to the following basic standardization for non-English text…

As a person who also lacks linguistic knowledge, I admire the straightforwardness of just going “Yeah, we have no idea, so we just won’t bother”. However, they seem to be pretty accurate in their self-assessment, because the normalization they end up with still has a glaring issue.

For a general ASR model like Whisper a rule of thumb is that it’s far better to train on raw data without any normalization: the model can learn to predict peripheral language features such as capitalization and punctuation given enough training data, and so a denormalization step is not required in actual application. Evaluation, however, is done with normalized text so that a dataset’s specific idiosyncrasies don’t reflect in the reported evaluation metric. In testing out the normalization function used by OpenAI, I plugged in some Malayalam text thinking I’d get basically the same thing back minus the punctuation:

ഞാൻ കേൾക്കാൻ ആഗ്രഹിക്കുന്ന വാക്കുകൾ കൊണ്ട് ഞാൻ എപ്പോഴും ജനങ്ങളെ ആശ്വസിപ്പിക്കാൻ ശ്രമിക്കുന്നു

Instead, I got this abomination:

ഞ ൻ ക ൾക ക ൻ ആഗ രഹ ക ക ന ന വ ക ക കൾ ക ണ ട ഞ ൻ എപ പ ഴ ജനങ ങള ആശ വസ പ പ ക ക ൻ ശ രമ ക ക ന ന

The culprit is pretty easy to find, right after the aforementioned passage:

Replace any markers, symbols, and punctuation characters with a space, i.e. when the Unicode category of each character in the NFKC-normalized string starts with M, S, or P

Unicode categories that start with P are indeed punctuations, and those that start with S are indeed symbols (currency, math, etc.), and it makes sense to remove both. Marks are a lot less obvious. There are three mark categories: Spacing Mark (Mc), Enclosing Mark (Me), and Nonspacing Mark (Mn). Enclosing marks consists of a dozen or so symbols that, well, enclose other characters. Most renderers don’t even support it, and a quick visual inspection is enough to convince anyone that these characters can be safely discarded. Spacing marks and nonspacing marks, however, are almost completely composed of diacritics in various languages, chiefly Indo-Aryan and Dravidian. Removing these characters and replacing them with whitespace usually gets you gibberish.

A direct look at their code reveals a direr state of affairs. By default, the normalizer calls a function remove_symbols that removes all characters whose Unicode category starts with M, S, or P, consistent with the paper:

def remove_symbols(s: str):
	"""
	Replace any other markers, symbols, punctuations with a space, keeping diacritics
	"""
	return "".join(
	" " if unicodedata.category(c)[0] in "MSP" else c for c in unicodedata.normalize("NFKC", s)
	)

However, if you set the parameter remove_diacritics to True, you instead get:

def remove_symbols_and_diacritics(s: str, keep=""):
	"""
	Replace any other markers, symbols, and punctuations with a space,
	and drop any diacritics (category 'Mn' and some manual mappings)
	"""
	return "".join(
		c
		if c in keep
		else ADDITIONAL_DIACRITICS[c]
		if c in ADDITIONAL_DIACRITICS
		else ""
		if unicodedata.category(c) == "Mn"
		else " "
		if unicodedata.category(c)[0] in "MSP"
		else c
		for c in unicodedata.normalize("NFKD", s)
)

Here, they seem to realize that marks indeed include diacritics, and they omit nonspacing marks instead of replacing it with a whitespace. This is actually marginally better than what remove_symbols does to diacritics, and this points to the the fundamental blunder being more “we didn’t proofread our code” and less “we didn’t check what Unicode categories meant”.

As we see before, OpenAI admits their normalization for anything non-English is barebones. Particularly for languages with no clear demarcation between words, it splits sentences into a sequence of characters, functionally measuring the CER (character error rate) instead of the reported WER (word error rate). This makes sense as a quick and clean solution: their numbers are measuring the wrong thing, but they acknowledge it. However, nowhere in the paper do they acknowledge that they do the same thing to a laundry list of languages that do have a clear demarcation between words: the same one that every other modern language uses. In the end, what you functionally get after normalization is a Frankenstein between CER and WER for languages unlucky enough to be using a particular class of diacritics.

Since this is the normalization they used to get their reported results, I reevaluated the medium model on the FLEURS dataset for all potentially affected languages with normalization that doesn’t just fling away diacritics to see if there was any real difference. Sure enough:

Every single language had a deflated WER, with the exception of Sindhi (not really sure what’s going on there). Granted, Whisper did hopelessly terrible in most of these languages to begin with (WER over 100% not really a good look), but Tamil, Hindi, and Urdu strike me as three languages where the discrepancy does make a difference: 25%ish is a manageable WER, 55% less so.

This becomes even more of an issue when people keep using the same damn normalization function for finetuned model evaluation, like in the Hugging Face Whisper Finetune Sprint, where the leaderboard for these languages is basically curated misinformation.

An easy out would be to use the CER: this makes everything more or less consistent for all languages and prevents issues with scripto continua writing systems and agglutinative languages. That said, this is a pervasive issue with non-English NLP research in general, where evaluation and its metrics are optimized exclusively for English, but I digress. Next time around, I hope OpenAI uses all the money they saved from underpaying and mentally scarring Kenyan workers to hire a linguist, or at very least, a proofreader.

All my friends who started learning Japanese after I did tapped out pretty quickly. Flimsy motivations notwithstanding, they also tapped out at the first real challenge in the entire process: cramming hundreds of words into your head as a baseline before you actually start immersing in the language. While I imagine this process is universal across all languages, Japanese also has the additional issue of the Kanji (漢字) system.

Japanese has two syllabaries: Hiragana (ひらがな) and Katakana (カタカナ). They’re pretty simple: 46 characters each, and they’re used to write out the sounds of the language. Hiragana is used for native Japanese words, and Katakana is used for foreign words. Kanji are the 10 thousand or so Chinese characters that form the third fundamental part of the Japanese writing system. They’re a necessary evil: a lack of word boundaries means you can’t read the language otherwise, and even if you put in spaces between words completely written in Hiragana,Nintendo games and other children's media do this, for the kiddies who are learning. it’s still much harder to read than if you just knew the Kanji, because of Japanese’s restricted syllabary and the homophones that result.

Learning Kanji is an arduous task, and as such there have been a lot of debate in the community as to the most efficient way to cram in these initially inexplicable moonrunes. One common way is to build it up from raw materials: kanji generally are made of radicals, and the meaning of each kanji is sometimes associated with the radicals it’s made of. A natural extension of this, and the method I used initially, is the Remembering The Kanji (RTK) method.

The RTK Method associates each Kanji with a mnemonic based on the radicalsI said radicals here for simplicity, but the actual radicals and the defined "components" the RTK method uses are different. it’s made of. For example, the Kanji for “tree” (木) is made of the radicals for “tree” and “one” (一), and the mnemonic for it is “A tree is one thing”. Much like all mnemonics, the idea is that the mnemonic is easier to remember than the actual Kanji, and that the mnemonic will eventually be replaced by the actual meaning of the Kanji.

That doesn’t happen.

It just doesn’t.

Mnemonics are a practically unnecessary crutch

Say you hit a Kanji which you don’t immediately recognize, but which you’ve “learnt” via the RTK method. You now have to pause reading, remember the mnemonic, go through it in your headProbably some garbage like this mnemonic for 喻, for instance: "A meeting of butchers is a perfect metaphor for the supposed secret talks(mouth) between Hitler and Stalin", and finally remember what the kanji meant. Fun fact! That took you 10 seconds, and you still don’t know how to read it, because the mnemonic only taught you the meaning, not how to pronounce it. How do you learn the pronunciation? Most Kanji have multiple pronounciations (usually called readings), so generally the idea is to just learn it on a word-by-word basis. So why not just learn the damn words in the first place?

This is my core issue with mnemonics. They’re a crutch that’s functionally unnecessary if you’re actually going to be using the information you’re trying to remember. Recall is only useful if it’s fast. Slow recall may as well be equivalent to not knowing the information at all, because guess what? We all have the Library of Babel in our pockets. We have the All-Knowning Information Box just sitting there, and we all have the muscle memory to look things up faster than we could spit out the damn mnemonic.

The only time mnemonics are useful is when you’re in an environment where you can’t look things up, and it’s sort of fine to take your time to remember it. I posit that most of these environments are either contrived or outdated: the most egregious offender being exams. They should be open book anyway if you actually want to test the material in any rigorous fashion; If your exam can be aced if you have your notes with you, then it isn’t a very good exam to begin with. The only other situations I can think of is something like surgery, where you might not have the luxury of whipping out your phone during someone’s heart transplant, but could spare the seconds to mnemonic out the rib bones or whatever. Still, I don’t know about you but I definitely wouldn’t want my surgeon to be relying on a mnemonic to remember where the liver is.

Mnemonics replace understanding

In some cases, you shouldn’t be memorizing at all. Once you actually understand the topic, whatever you’re trying to remember becomes trivial as you figure out the underlying structure. Education that focuses on memorization over understanding is doomed to fail.

As someone who went to school in India, this is a reoccurring cliche across basically every single subject. But even for educational systems that aren’t utter dogshit, there still exists remnants. Mnemonics are a notable canary in the coal mine.

Let’s take trignometry. Two well-known mnemonics here: SOH-CAH-TOA and All Students Take Calculus.

SOH-CAH-TOA

A nice clean mnemonic for remembering how the basic trigonometric functions relate to the sides of a triangle:

Sine = Opposite ÷ Hypotenuse

Cosine = Adjacent ÷ Hypotenuse

Tangent = Opposite ÷ Adjacent

Now, this one is bad on the face of it. Trigonometry is used so consistently in your school education that needing a mnemonic to remember this is unnecessary by itself. However, this framing also fundamentally misunderstands the trigonometric functions. They are not about triangles, they’re about circles.

Indeed, the only reason we usually care about the trigonometric functions is because for a particular point on the circle, its coordinates \((x, y)\) is \((\cosθ,\sinθ)\).

It is in this context where trigonometry is actually useful. This perspective is the fundamental to further understanding down the line as trigonometry pops up again and again in different contexts. 3Blue1Brown makes this point way better than I can in this video where he teaches trigonometry fundamentals, so give that a watch.

Now, while \(\sin\) and \(\cos\) are the main two, all trigonometric functions are indeed foundationally mapped on the circle:

\(\cot\), \(\csc\) and \(\cot\) are for the most part remnants of an era where math’s most practical application was marine navigation. \(\tan\) is, well, the tangent, so being the length of the tangent where it intersects the x-axis makes sense. Practically, you don’t even need to remember what it represents, just how it relates to \(\sin\) and \(\cos\).

What you’re left to memorize then is that \(\sin\theta\) is \(y\), \(\cos\theta\) is \(x\), and \(\tan\) is \(\frac{\sin\theta}{\cos\theta}\). For circles of a non-unit radius \(r\), the functions would scale too. Considering \(r\) is actually the hypotenuse of the triangle in question, one can derive pretty easily the relations with the sides.

Now, you might complain that this isn’t that big of a reduction in memory. Indeed, with the mnemonic, it’s true that the original is probably easier to immediately cram into your head. However the original perspective is antithetical to an actual understanding of what you’re doing, and makes things harder later down the line. Case in point: the quadrants.

All Students Take Calculus (or any of its million variations)

The Cartesian plane can be subdivided into quadrants depending on the sign of the axis, and in each quadrant the signs of the trigonometric functions vary:

• Quadrant I (angles from 0 to 90 degrees, or 0 to π/2 radians): All trigonometric functions are positive in this quadrant.
• Quadrant II (angles from 90 to 180 degrees, or π/2 to π radians): Sine and cosecant functions are positive in this quadrant.
• Quadrant III (angles from 180 to 270 degrees, or π to 3π/2 radians): Tangent and cotangent functions are positive in this quadrant.
• Quadrant IV (angles from 270 to 360 degrees, or 3π/2 to 2π radians): Cosine and secant functions are positive in this quadrant. The following image uses All Science Teachers (are) Crazy but the concept is the same:

An innocuous tool to memorize mathematical facts, one might say. After all, how would one remember the signs of each trigonometric functions given their angle otherwise? Maybe, I don’t know, looking at where it is on the fucking circle.

The signs of \(x\) and \(y\), the defining feature of each quadrant directly give you the signs of \(\sin\) and \(\cos\), and deriving the sign of the rest of the trigonometric functions still takes less time than reciting that dumbass mnemonic.

Many of the conversions in trigonometry become intuitive and quick when this understanding kicks in. Why is \(\sin\theta = -\sin(2\pi - \theta)\)? Just look at the circle! \(\sin\theta = \cos(\frac{\pi}{2} - \theta)\)? Keep in mind the axes, and summon the circle! Want to visualize the function? Rotate around the goddamn circle!

It’s frankly a disgrace: students are given a dumb mnemonic that both robs proper learning and makes things harder to remember in the end. While the mnemonic itself is not necessarily the root of the problem, it’s indicative of a fundamentally incurious form of learning; one that would rather memorize than understand. That’s just sad. Nobody should have to remember All Students Take Calculus.

Final Notes

I imagine a lot of the members in the Memorization community will have several qualms about the opinions I’ve expressed here. So I’ll temper my piece with this: mnemonics are good at what they’re designed to do. They are uniquely effective in letting you memorize (given leeway in terms of recall time) unconnected, disjointed information in large quantities. There’s a reason why literally every memory sports champion uses some form of mnemonics, and it’s the ideal way of learning e.g. the digits of \(\pi\), for instance. I’ll give that to you, mnemonics: you work. I just don’t think you’re very useful.