Oven logo

Oven

srsly2.4.8

Published

Modern high-performance serialization utilities for Python

pip install srsly

Package Downloads

Weekly DownloadsMonthly Downloads

Project URLs

Requires Python

>=3.6

srsly: Modern high-performance serialization utilities for Python

This package bundles some of the best Python serialization libraries into one standalone package, with a high-level API that makes it easy to write code that's correct across platforms and Pythons. This allows us to provide all the serialization utilities we need in a single binary wheel. Currently supports JSON, JSONL, MessagePack, Pickle and YAML.

tests PyPi conda GitHub Python wheels

Motivation

Serialization is hard, especially across Python versions and multiple platforms. After dealing with many subtle bugs over the years (encodings, locales, large files) our libraries like spaCy and Prodigy had steadily grown a number of utility functions to wrap the multiple serialization formats we need to support (especially json, msgpack and pickle). These wrapping functions ended up duplicated across our codebases, so we wanted to put them in one place.

At the same time, we noticed that having a lot of small dependencies was making maintenance harder, and making installation slower. To solve this, we've made srsly standalone, by including the component packages directly within it. This way we can provide all the serialization utilities we need in a single binary wheel.

srsly currently includes forks of the following packages:

Installation

⚠️ Note that v2.x is only compatible with Python 3.6+. For 2.7+ compatibility, use v1.x.

srsly can be installed from pip. Before installing, make sure that your pip, setuptools and wheel are up to date.

python -m pip install -U pip setuptools wheel
python -m pip install srsly

Or from conda via conda-forge:

conda install -c conda-forge srsly

Alternatively, you can also compile the library from source. You'll need to make sure that you have a development environment with a Python distribution including header files, a compiler (XCode command-line tools on macOS / OS X or Visual C++ build tools on Windows), pip and git installed.

Install from source:

# clone the repo
git clone https://github.com/explosion/srsly
cd srsly

# create a virtual environment
python -m venv .env
source .env/bin/activate

# update pip
python -m pip install -U pip setuptools wheel

# compile and install from source
python -m pip install .

For developers, install requirements separately and then install in editable mode without build isolation:

# install in editable mode
python -m pip install -r requirements.txt
python -m pip install --no-build-isolation --editable .

# run test suite
python -m pytest --pyargs srsly

API

JSON

📦 The underlying module is exposed via srsly.ujson. However, we normally interact with it via the utility functions only.

function srsly.json_dumps

Serialize an object to a JSON string. Falls back to json if sort_keys=True is used (until it's fixed in ujson).

data = {"foo": "bar", "baz": 123}
json_string = srsly.json_dumps(data)
ArgumentTypeDescription
data-The JSON-serializable data to output.
indentintNumber of spaces used to indent JSON. Defaults to 0.
sort_keysboolSort dictionary keys. Defaults to False.
RETURNSstrThe serialized string.

function srsly.json_loads

Deserialize unicode or bytes to a Python object.

data = '{"foo": "bar", "baz": 123}'
obj = srsly.json_loads(data)
ArgumentTypeDescription
datastr / bytesThe data to deserialize.
RETURNS-The deserialized Python object.

function srsly.write_json

Create a JSON file and dump contents or write to standard output.

data = {"foo": "bar", "baz": 123}
srsly.write_json("/path/to/file.json", data)
ArgumentTypeDescription
pathstr / PathThe file path or "-" to write to stdout.
data-The JSON-serializable data to output.
indentintNumber of spaces used to indent JSON. Defaults to 2.

function srsly.read_json

Load JSON from a file or standard input.

data = srsly.read_json("/path/to/file.json")
ArgumentTypeDescription
pathstr / PathThe file path or "-" to read from stdin.
RETURNSdict / listThe loaded JSON content.

function srsly.write_gzip_json

Create a gzipped JSON file and dump contents.

data = {"foo": "bar", "baz": 123}
srsly.write_gzip_json("/path/to/file.json.gz", data)
ArgumentTypeDescription
pathstr / PathThe file path.
data-The JSON-serializable data to output.
indentintNumber of spaces used to indent JSON. Defaults to 2.

function srsly.write_gzip_jsonl

Create a gzipped JSONL file and dump contents.

data = [{"foo": "bar"}, {"baz": 123}]
srsly.write_gzip_json("/path/to/file.jsonl.gz", data)
ArgumentTypeDescription
pathstr / PathThe file path.
lines-The JSON-serializable contents of each line.
appendboolWhether or not to append to the location. Appending to .gz files is generally not recommended, as it doesn't allow the algorithm to take advantage of all data when compressing - files may hence be poorly compressed.
append_new_lineboolWhether or not to write a new line before appending to the file.

function srsly.read_gzip_json

Load gzipped JSON from a file.

data = srsly.read_gzip_json("/path/to/file.json.gz")
ArgumentTypeDescription
pathstr / PathThe file path.
RETURNSdict / listThe loaded JSON content.

function srsly.read_gzip_jsonl

Load gzipped JSONL from a file.

data = srsly.read_gzip_jsonl("/path/to/file.jsonl.gz")
ArgumentTypeDescription
pathstr / PathThe file path.
RETURNSdict / listThe loaded JSONL content.

function srsly.write_jsonl

Create a JSONL file (newline-delimited JSON) and dump contents line by line, or write to standard output.

data = [{"foo": "bar"}, {"baz": 123}]
srsly.write_jsonl("/path/to/file.jsonl", data)
ArgumentTypeDescription
pathstr / PathThe file path or "-" to write to stdout.
linesiterableThe JSON-serializable lines.
appendboolAppend to an existing file. Will open it in "a" mode and insert a newline before writing lines. Defaults to False.
append_new_lineboolDefines whether a new line should first be written when appending to an existing file. Defaults to True.

function srsly.read_jsonl

Read a JSONL file (newline-delimited JSON) or from JSONL data from standard input and yield contents line by line. Blank lines will always be skipped.

data = srsly.read_jsonl("/path/to/file.jsonl")
ArgumentTypeDescription
pathstr / PathThe file path or "-" to read from stdin.
skipboolSkip broken lines and don't raise ValueError. Defaults to False.
YIELDS-The loaded JSON contents of each line.

function srsly.is_json_serializable

Check if a Python object is JSON-serializable.

assert srsly.is_json_serializable({"hello": "world"}) is True
assert srsly.is_json_serializable(lambda x: x) is False
ArgumentTypeDescription
obj-The object to check.
RETURNSboolWhether the object is JSON-serializable.

msgpack

📦 The underlying module is exposed via srsly.msgpack. However, we normally interact with it via the utility functions only.

function srsly.msgpack_dumps

Serialize an object to a msgpack byte string.

data = {"foo": "bar", "baz": 123}
msg = srsly.msgpack_dumps(data)
ArgumentTypeDescription
data-The data to serialize.
RETURNSbytesThe serialized bytes.

function srsly.msgpack_loads

Deserialize msgpack bytes to a Python object.

msg = b"\x82\xa3foo\xa3bar\xa3baz{"
data = srsly.msgpack_loads(msg)
ArgumentTypeDescription
databytesThe data to deserialize.
use_listboolDon't use tuples instead of lists. Can make deserialization slower. Defaults to True.
RETURNS-The deserialized Python object.

function srsly.write_msgpack

Create a msgpack file and dump contents.

data = {"foo": "bar", "baz": 123}
srsly.write_msgpack("/path/to/file.msg", data)
ArgumentTypeDescription
pathstr / PathThe file path.
data-The data to serialize.

function srsly.read_msgpack

Load a msgpack file.

data = srsly.read_msgpack("/path/to/file.msg")
ArgumentTypeDescription
pathstr / PathThe file path.
use_listboolDon't use tuples instead of lists. Can make deserialization slower. Defaults to True.
RETURNS-The loaded and deserialized content.

pickle

📦 The underlying module is exposed via srsly.cloudpickle. However, we normally interact with it via the utility functions only.

function srsly.pickle_dumps

Serialize a Python object with pickle.

data = {"foo": "bar", "baz": 123}
pickled_data = srsly.pickle_dumps(data)
ArgumentTypeDescription
data-The object to serialize.
protocolintProtocol to use. -1 for highest. Defaults to None.
RETURNSbytesThe serialized object.

function srsly.pickle_loads

Deserialize bytes with pickle.

pickled_data = b"\x80\x04\x95\x19\x00\x00\x00\x00\x00\x00\x00}\x94(\x8c\x03foo\x94\x8c\x03bar\x94\x8c\x03baz\x94K{u."
data = srsly.pickle_loads(pickled_data)
ArgumentTypeDescription
databytesThe data to deserialize.
RETURNS-The deserialized Python object.

YAML

📦 The underlying module is exposed via srsly.ruamel_yaml. However, we normally interact with it via the utility functions only.

function srsly.yaml_dumps

Serialize an object to a YAML string. See the ruamel.yaml docs for details on the indentation format.

data = {"foo": "bar", "baz": 123}
yaml_string = srsly.yaml_dumps(data)
ArgumentTypeDescription
data-The JSON-serializable data to output.
indent_mappingintMapping indentation. Defaults to 2.
indent_sequenceintSequence indentation. Defaults to 4.
indent_offsetintIndentation offset. Defaults to 2.
sort_keysboolSort dictionary keys. Defaults to False.
RETURNSstrThe serialized string.

function srsly.yaml_loads

Deserialize unicode or a file object to a Python object.

data = 'foo: bar\nbaz: 123'
obj = srsly.yaml_loads(data)
ArgumentTypeDescription
datastr / fileThe data to deserialize.
RETURNS-The deserialized Python object.

function srsly.write_yaml

Create a YAML file and dump contents or write to standard output.

data = {"foo": "bar", "baz": 123}
srsly.write_yaml("/path/to/file.yml", data)
ArgumentTypeDescription
pathstr / PathThe file path or "-" to write to stdout.
data-The JSON-serializable data to output.
indent_mappingintMapping indentation. Defaults to 2.
indent_sequenceintSequence indentation. Defaults to 4.
indent_offsetintIndentation offset. Defaults to 2.
sort_keysboolSort dictionary keys. Defaults to False.

function srsly.read_yaml

Load YAML from a file or standard input.

data = srsly.read_yaml("/path/to/file.yml")
ArgumentTypeDescription
pathstr / PathThe file path or "-" to read from stdin.
RETURNSdict / listThe loaded YAML content.

function srsly.is_yaml_serializable

Check if a Python object is YAML-serializable.

assert srsly.is_yaml_serializable({"hello": "world"}) is True
assert srsly.is_yaml_serializable(lambda x: x) is False
ArgumentTypeDescription
obj-The object to check.
RETURNSboolWhether the object is YAML-serializable.