What tokenizer does GPT-4 use?

Question

Accepted Answer

GPT-4 and GPT-4o use the cl100k_base tokenizer. This is the same tokenizer used by GPT-3.5-turbo. Token counts are identical between these models, but pricing differs.

GPT-4 Token Counter

Frequently Asked Questions