Datasets · AI Nav City

Core facts

Read by a script from the official API, not rewritten by a model.

Repository: https://github.com/huggingface/datasets
License: Apache-2.0
Languages: Python
Maintainer: huggingface
Homepage: https://huggingface.co/docs/datasets
Verified: 2026-11-02

What it is

Datasets is a library providing two main features. One of these two features is one-line dataloaders for many public datasets. Datasets can load datasets based on versioned git repositories maintained by the dataset authors.

The library provides two main dataset classes. For reproducibility reasons, the project asks users to pin the revision of the repositories they use.

Main features

Datasets has native support for CSV, JSON, JSONL, Parquet, Arrow, XML, Text, Webdataset, and more formats.
Datasets has built-in support for text, audio, image, video, PDF, and NIfTI (3D medical) data.
If a dataset is bigger than the disk, or if one does not want to wait to download the data, streaming mode can be used.
Datasets can read and write directly from and to Hugging Face Storage Buckets for mutable, large-scale raw data.
Datasets can load and process AI agent traces (prompts, tool calls, responses) from the Hub.
Datasets has built-in FAISS and Elasticsearch index support for similarity search.
Datasets supports flexible JSON and structured data through its Json feature type.

Getting started

Datasets can be installed from PyPi and should be installed in a virtual environment, for instance venv or conda.
Datasets supports various optional features via extras.
The Datasets API is centered around a single function that instantiates a dataset.
One-line dataset loading loads AI-ready datasets from the Hugging Face Hub or local files.

From the README · bash

pip install datasets

From the README · bash

pip install "datasets @ git+https://github.com/huggingface/datasets.git"

From the README · bash

conda install -c huggingface -c conda-forge datasets

From the README · bash

# For audio (torchcodec)
pip install datasets[audio]

# For image/video (Pillow, torchcodec)
pip install datasets[vision]

# For PDFs/NIfTI (pdfplumber, nibabel)
pip install datasets[pdfs,nibabel]

# For PyTorch/TensorFlow/JAX integration
pip install datasets[torch,tensorflow,jax]

From the README · python

from datasets import load_dataset

# Load a dataset and print the first example in the training set
squad_dataset = load_dataset('rajpurkar/squad')
print(squad_dataset['train'][0])

# Process the dataset - add a column with the length of the context texts
dataset_with_length = squad_dataset.map(lambda x: {"length": len(x["context"])})

# Tokenize the context texts (using a tokenizer from the 🤗 Transformers library)
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')

tokenized_dataset = squad_dataset.map(lambda x: tokenizer(x['context']), batched=True)

# Tokenize chat conversations with a chat template (using a model that supports chat templates)
# This is useful for fine-tuning instruction/chat models

# Load a popular chat dataset (ultrachat_200k contains ~200k AI assistant conversations)
chat_dataset = load_dataset('HuggingFaceH4/ultrachat_200k', split='train_sft')

chat_tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen2.5-7B-Instruct')

def tokenize_chat(examples):
    # Apply the chat template and tokenize in one step
    return chat_tokenizer.apply_chat_template(examples["messages"])

tokenized_chat_dataset = chat_dataset.map(tokenize_chat, batched=True)

Who it is for

The Dataset class is an in-memory / memory-mapped dataset backed by Apache Arrow. The Dataset class supports indexing, slicing, random access and caching. The IterableDataset class is a lazy, streamable dataset for large-scale / out-of-core processing.

The IterableDataset class supports streaming and infinite iteration. Dataset and IterableDataset are wrapped in DatasetDict / IterableDatasetDict for multi-split datasets, for example train/test/val. Datasets has native conversion to and from NumPy, Pandas, Polars, Arrow, PyTorch, TensorFlow, JAX, and Spark.

The public datasets behind the one-line dataloaders include text datasets in 467 languages and dialects. There is a page on how to upload a dataset to the Hub using a web browser or Python. There is a page on how to upload a dataset using Git.

The project uses Ruff in its code style guidelines. A dataset owner who does not want the dataset included in the Hugging Face Hub can get in touch by opening a discussion or a pull request in the Community tab of the dataset page.

Related tools listed on this site.