pypi i pystream-protobuf


Python implementation of stream library for streaming google protocol buffer objects

by Ali Ghaffaari

1.6.3 (see all)License:MIT
pypi i pystream-protobuf

Build Status PyPI Release PyPI Status Python License


Python implementation of stream library.


This library enables stream processing of protobuf messages (or any serializable objects since v1.6.3); i.e. multiple protobuf messages can be written/read into/from a single stream or file.

It was originally developed to parse/write vg file formats (.vg, .gam, etc). However, it can be used for any arbitrary protocol buffer messages.

Refer to the C++ stream library for more details.


@vg users: The new version of stream library, now as a part of libvgio, writes a header at the start of the stream depending on the output format. For example, headers like b'GAM' or b'VG' can be found before the actual protobuf messages in GAM and VG files repectively. In this case, you should provide the expected value using header keyword argument; e.g. stream.parse('file.gam', vg_pb2.Alignment, header=b'GAM') for GAM files (since version v1.6.2).


The encoding is simple. Messages are written in groups of different sizes. Each group starts with its size; i.e. the number of messages in that group. Then, the size of each message is followed by the encoded message itself. Quoted from Google Protobuf Developer Guide:

The Protocol Buffer wire format is not self-delimiting, so protocol buffer parsers cannot determine where a message ends on their own. The easiest way to solve this problem is to write the size of each message before you write the message itself. When you read the messages back in, you read the size, then read the bytes into a separate buffer, then parse from that buffer.

By default, the stream is considered compressed by GZip. However, uncompressed stream processing is possible by passing gzip=False to any API calls.


You can install pyStream using pip:

pip install pystream-protobuf


See Wiki for usage documentation.


In case, you work with the source code and need to build the package:

python setup.py build

The proto files in the test module required to be compiled before running test cases. To do so, it is required to have Google protobuf compiler (>=3.0.2) installed. After installing protobuf compiler, run:

make init

to compile proto files required for test module and then:

make test

to run tests.

2yrs ago
2yrs ago
2yrs ago
3yrs ago
No alternatives found
No tutorials found
Add a tutorial
No dependencies found

Rate & Review

No reviews found
Be the first to rate