All Collections Support

Node.js

Cross-Platform

Linux

Containers
eBPF
Arch-based Projects
AppImage
Omarchy

macOS

Screensavers
Apps
Open Source Apps

Windows

PowerToys Run Plugins

.NET

Core
Roslyn

JavaScript

Promises
Standard Style
Must Watch Talks
Tips
Network Layer
Micro npm Packages
Mad Science npm Packages
Maintenance Modules
npm
AVA
ESLint
Functional Programming
Observables
npm scripts
30 Seconds of Code
Ponyfills

Swift

Education
Playgrounds

Python

Asyncio
Scientific Audio
CircuitPython
Data Science
Typing
MicroPython

Rust

Pest

Scala

Scala Native

R

Learning

Common Lisp

Learning

Java

RxJava
J2ME

PHP

Composer

CSS

Critical-Path Tools
Scalability
Must-Watch Talks
Protips
Frameworks

React

Relay
React Hooks

jQuery

Tips

Flask
Docker
Vagrant
Pyramid
Play1 Framework
CakePHP
Symfony
Laravel
Rails
Phalcon
Useful `.htaccess` Snippets
nginx
Dropwizard
Kubernetes
Lumen
Serverless Framework
Apache Wicket
Vert.x
Terraform
Vapor
Dash
FastAPI
CDK
IAM
Slim
Fiber
Kustomize
OpenTofu
Reflex

Symfony

Education

Laravel

Education
TALL Stack

Rails

Gems

University Courses
Data Science
Machine Learning
Speech and Natural Language Processing
Linguistics
Cryptography
Computer Vision
Deep Learning
Deep Vision
Open Source Society University
Functional Programming
Empirical Software Engineering
Static Analysis & Code Quality
Information Retrieval
Quantum Computing
Theoretical Computer Science
Conversational AI
Generative AI
Position-Based Quantum Cryptography

Data Science

Tutorials

Machine Learning

Tutorials
ML with Ruby
Core ML Models
H2O
Software Engineering for Machine Learning
AI in Finance
JAX
XAI

Speech and Natural Language Processing

Spanish
NLP with Ruby
Question Answering
Natural Language Generation

Cryptography

Papers

Deep Learning

TensorFlow
TensorFlow.js
TensorFlow Lite
Papers
Education

Big Data
Public Datasets
Hadoop
Data Engineering
Streaming
Apache Spark
Qlik
Splunk
Network Analysis

Papers We Love
Talks
Algorithms
Algorithm Visualizations
Artificial Intelligence
Search Engine Optimization
Competitive Programming
Math
Recursion Schemes
Audit Algorithms
AGI & CoCoSci
Complex Systems
VLM Architectures

Algorithms

Education

Free Programming Books
Go Books
R Books
Mind Expanding Books
Book Authoring
Elixir Books

Sublime Text
Vim
Neovim
Emacs
Atom
Visual Studio Code

Game Development
Game Talks
Godot
Open Source Games
Unity
Chess
LÖVE
PICO-8
Game Boy Development
Construct 2
Gideros
Minecraft
ComputerCraft
Game Datasets
Haxe Game Development
libGDX
PlayCanvas
Game Remakes
Flame
Discord Communities
CHIP-8
Games of Coding
Esports
Learn Gamedev
Game Engine Development
GameMaker
Game Production
Babylon.js
Roblox

Quick Look Plugins
Dev Env
Dotfiles
Shell
Fish
Command-Line Apps
ZSH Plugins
GitHub
Git Cheat Sheet & Git Flow
Git Tips
Git Add-ons
Git Hooks
SSH
FOSS for Developers
Hyper
PowerShell
Alfred Workflows
Terminals Are Sexy
GitHub Actions
WezTerm

GitHub

Browser Extensions
Cheat Sheet
Pinned Gists

Science Fiction
Fantasy
Podcasts
Email Newsletters
IT Quotes

Database
MySQL
SQLAlchemy
InfluxDB
Neo4j
MongoDB
RethinkDB
TinkerPop
PostgreSQL
CouchDB
HBase
NoSQL Guides
Database Tools
TypeDB
Cassandra
TDengine
Supabase
PocketBase
Neon

Fonts
Codeface
Stock Resources
GIF
Music
Open Source Documents
Audio Visualization
Broadcasting
Pixel Art
FFmpeg
Icons
Audiovisual
VLC
Audio Over IP

CLI Workshoppers
Learn to Program
Speaking
Tech Videos
Dive into Machine Learning
Computer History
Programming for Kids
STEAM
Educational Games
JavaScript Learning
CSS Learning
Product Management
Roadmaps
YouTubers
Hackathon
Certificates

Application Security
Security
CTF
Cyber Security University
Malware Analysis
Android Security
Hacking
Hacking Spots
Honeypots
Incident Response
Vehicle Security and Car Hacking
Web Security
Lockpicking
Cybersecurity Blue Team
Fuzzing
Embedded and IoT Security
GDPR
DevSecOps
Executable Packing
Malware Persistence
EVM Security
Password Cracking
Security Card Games
Suricata
Prompt Injection
Detection Engineering
Annual Security Reports
CI/CD Attacks
OpenID Connect

Umbraco
Refinery CMS
Wagtail
Textpattern
Drupal
Craft CMS
Sitecore
Silverstripe CMS
Directus
Plone
Payload

Robotics
Internet of Things
Electronics
Bluetooth Beacons
Electric Guitar Specifications
Plotters
Robotic Tooling
LIDAR
Open Hardware
ADS-B
Flying FPV

Open Companies
Places to Post Your Startup
OKR Methodology
Indie
Tools of the Trade
Clean Tech
Wardley Maps
Social Enterprise
Engineering Team Management
Developer-First Products
Billing
Engineering Strategy

Slack
Remote Jobs
Productivity
Niche Job Boards
Programming Interviews
Code Review
Creative Technology
Internships

Slack

Communities

Software-Defined Networking
PCAPTools
Real-Time Communications
SNMP
Scapy
Cilium

Bitcoin
Ripple
Non-Financial Blockchain
Mastodon
Ethereum
Blockchain AI
EOSIO
Corda
Waves
Substrate
Golem
Stacks
Algorand
ZeroNet
Cosmos SDK
Tor
ATProto

Biomedical Information Extraction
Computational Neuroscience
Diversity
Digital History
Empathy in Engineering
Healthcare
Humane Technology
Mental Health
Neuroscience
Digital Humanities
Lucid Dreams
Neuroimaging
Transgender

Creative Tech Events
Events in Italy
Events in the Netherlands

Testing
Visual Regression Testing
Selenium
Appium
TAP
JMeter
k6
Playwright
Quality Assurance Roadmap
Gatling
CodeRabbit

JSON

GeoJSON
Datasets

AMA

Answers

Unicode

Code Points

project awesome

♥ ♥ Support us

Type at least 2 characters to search

Home / Machine Learning / Natural Language Processing > huggingface/tokenizers

Natural Language Processing > huggingface/tokenizers

Fast State-of-the-Art Tokenizers optimized for Research and Production

Package 10.6k stars GitHub

Back to Machine Learning