Append-only, with snapshots
Records are appended and snapshotted, so a dataset can be read as it stood at a point in time.
Dataset storage
datashard
Table format for datasets and blobs, without standing up a database to hold them.
Available
What it is
Dataset and blob storage with the properties an Iceberg table gives you — append-only records, snapshots, time travel and Parquet on disk or S3 — as a Python library rather than a service. It replaces raw pickle, CSV and JSON dumps, hand-managed Parquet folders, and the habit of standing up a database purely to hold blobs.
Who it is for. Python projects that persist recordings, datasets or analytical data and need versioned, readable storage rather than a folder of files nobody can date.
- Python
- Parquet
- Snapshots
- Time travel
- S3
What it does
Disk or S3
The same table format on a local disk or in object storage, so moving between them is a configuration change.
Parquet underneath
Data lands as Parquet, readable by anything that reads Parquet, rather than in a format only this library understands.
Ready to look at datashard?
For the documentation, a pilot or an integration, talk to us directly.