ttok 1.0 switches default tokenizer
What you need to know
Simon Willison released ttok 1.0, changing the token-counting CLI's default tokenizer from GPT-4 to the GPT-5/GPT-6 family. This follows ttok 0.4, released a day earlier, which added a --list-models command and fixed a Click warning. Willison says the switch follows an experiment by William Liu showing all seven GPT models tested shared the same tokenizer.
Generated by AI from the reporting · updated 2 days ago
Developments
2 developments
- Oct 9 00:34 · 1 reportsttok 1.0 released, defaults to GPT-5/GPT-6 tokenizerSimon Willison: ttok 1.0 released, defaults to GPT-5/GPT-6 tokenizer
- Oct 8 23:34 · 1 reportsttok 0.4 released with a --list-models commandSimon Willison: ttok 0.4 released with a --list-models command
Timeline
Follow the reports to see every side of the story.
- Simon Willisonttok 1.0 released, defaults to GPT-5/GPT-6 tokenizer
Simon Willison released ttok 1.0, switching the default tokenizer from GPT-4 to the GPT-5/GPT-6 family. The change follows an experiment by William Liu showing all seven GPT models (5.5.
- Simon Willisonttok 0.4 released with a --list-models command
Simon Willison released ttok 0.4, updating his tiktoken-based token-counting CLI after a couple of years. The release fixes a Click warning.
Heat trend for this story
Not enough continuous data yet to chart a trend.