What is a token? Tokenization, and why Hungarian costs more

Language models are priced per token, limited by tokens and measured in tokens per second. But a token isn’t a word or a character. It is a piece of text the model’s tokenizer has learned to treat as one unit, and how text is cut into those pieces decides how much an AI model costs you, how much of a document it can read at once, and how well it handles your language. We measured it for Hungarian: the same sentence takes about twice as many tokens as in English.

What a token is

A model can’t read letters directly. Before text reaches the network, the tokenizer splits it into tokens and replaces each with a number, its ID in a fixed vocabulary. The model reads and writes sequences of those IDs; the tokenizer turns its output back into text.

Common words are usually one token, often including the space before them. Rarer words are split into several pieces. Using OpenAI’s o200k_base tokenizer, the one GPT-4o uses:

Numbers, code, emoji and rare names are split up in the same way. Every model family has its own tokenizer, so the same text gives different counts on different models.

How tokenizers learn their pieces

Most modern tokenizers use a variant of byte-pair encoding (BPE), adapted for language models by Sennrich, Haddow and Birch (2016). It starts from single characters or bytes and repeatedly merges the pair that appears most often in a large sample of text into a new token, until the vocabulary reaches a set size, typically 100,000 to 250,000 entries.

The result reflects the sample. Strings that are frequent in the training text become single tokens; strings that are rare stay in pieces. Since that text is mostly English on most models, English words tend to get whole tokens and other languages get fragments. Nothing in BPE understands grammar: the pieces of házainkban above don’t line up with its real parts, ház-aink-ban (house-our-in).

How to estimate a token count

For English, OpenAI’s rule of thumb is about four characters, or three-quarters of a word, per token: 1,000 tokens is roughly 750 words. That rule doesn’t transfer to other languages, to code, or to tables of numbers. To know for sure:

• Use the provider’s tokenizer, or a token counter built on it, on a sample of your real text.
• Read the usage numbers that APIs return with every reply: input tokens and output tokens. In LE PEAK, they are under each reply.

Why some languages need more tokens

The same meaning costs a different number of tokens in different languages. Petrov et al. (2023) compared translations of the same texts and found tokenized lengths differing by up to 15 times between languages; Ahia et al. (2023) showed what that means for API bills. Two things drive it: how much of the language the tokenizer saw, and how the language builds words.

Hungarian is agglutinative: it stacks suffixes onto a stem, so one word carries what English spreads over several. Házainkban is “in our houses”. That makes Hungarian words long and varied, and a tokenizer trained mostly on English has seen few of them whole, so it breaks them into short pieces.

We measured it: Hungarian vs English

We took Article 1 of the Universal Declaration of Human Rights in English and in its official Hungarian translation, and counted tokens with two OpenAI tokenizers (via the open-source gpt-tokenizer library, version 2.9.0).

The Hungarian text has fewer words and about the same number of characters, yet takes roughly twice the tokens: about 2.6 tokens per word, against 1.1 in English. The newer tokenizer, with twice the vocabulary, narrows the gap but doesn’t close it. One sentence is a small sample, so treat the ratio as an illustration and measure your own text, but it matches what larger studies report for languages like Hungarian.

What it means in practice

For Hungarian text on a tokenizer like these, every token-based number gets worse by about the same factor:

• Cost. At a per-token price, the same content costs about twice as much in Hungarian as in English. See LLM API pricing explained.
• Context. A context window holds about half as much Hungarian text; see What is a context window?
• Speed. Models generate a fixed number of tokens a second, so a Hungarian reply of the same length takes longer.
• Quality. Fragmented words give the model a harder job, which is one reason general models stumble more on Hungarian, and legal Hungarian in particular. It is part of the case for models built around the language; see AI for Hungarian law.

KTwo hasn’t published the tokenizers of KTwo Crust 1 Beta or KTwo Mantle 0.5, so we can’t give their counts here.

Frequently asked questions

What is a token in AI?

A piece of text, often a word or part of a word, that a language model’s tokenizer treats as one unit. Models read, write, limit and price text in tokens.

How many words is 1,000 tokens?

About 750 words of English text, by OpenAI’s rule of thumb. Other languages, code and numbers usually take more tokens per word; Hungarian took about 2.6 tokens per word in our measurement.

Why do AI models charge per token?

Because the work a model does, and the memory it needs, grows with the number of tokens it reads and writes, not with words or characters.

Why does Hungarian use more tokens than English?

Tokenizers are trained mostly on English, and Hungarian builds long words from a stem and many suffixes, so its words are rarely in the vocabulary whole and get split into several pieces each.

Do all AI models count tokens the same way?

No. Every model family has its own tokenizer and vocabulary, so the same text gives different token counts, and different prices, on different models.