mypy-protobuf

open source tools to generate mypy stubs from protobufs

Showing:

Popularity

Downloads/wk

0

GitHub Stars

386

Maintenance

Last Commit

2d ago

Contributors

28

Package

Dependencies

2

License

Apache License 2.0

Categories

Readme

mypy-protobuf: Generate mypy stub files from protobuf specs

CI pypi

license

2.10 will be the last version of mypy-protobuf which supports targeting python 2.7. Next will be mypy-protobuf 3.0 which will auspiciously only support targeting python3

See Changelog for recent changes.

Requirements to run mypy-protobuf

protoc >= 3.18.0 python-protobuf >= 3.17.4 - matching protoc release python >= 3.6 - for running mypy-protobuf plugin.

Requirements to run mypy on stubs generated by mypy-protobuf

mypy >= v0.910 python-protobuf >= 3.18.0 - matching protoc release types-protobuf==3.17.4 - for stubs from the google.protobuf library

To run mypy on code generated with grpc plugin - you'll additionally need

grpcio>=1.40.0 grpcio-tools>=1.40.0 grpc-stubs>=1.24.7

Other configurations may work, but are not supported in testing currently. We would be open to expanding this list if a need arises - file an issue on the issue tracker.

Installation

The plugin can be installed with

pip3 install mypy-protobuf

To install unreleased

REV=main  # or whichever unreleased git rev you'd like
pip3 install git+https://github.com/dropbox/mypy-protobuf.git@$REV

# For older (1.x) versions of mypy protobuf - you may need
pip3 install git+https://github.com/dropbox/mypy-protobuf.git@$REV#subdirectory=python

In order to run mypy on the generated code, you'll need to install

pip3 install mypy>=0.910 types-protobuf>=0.1.14

Usage

On posix, protoc-gen-mypy is installed to python's executable bin. Assuming that's on your $PATH, you can run

protoc --python_out=output/location --mypy_out=output/location

Alternately, you can explicitly provide the path:

protoc --plugin=protoc-gen-mypy=path/to/protoc-gen-mypy --python_out=output/location --mypy_out=output/location

Check the version number with

> protoc-gen-mypy --version

Getting Help

Find other developers in the mypy-protobuf slack workspace (Invitation Link). If your company uses slack and mypy-protobuf, you may opt to use slack-connect to make a shared channel.

Implementation

The implementation of the plugin is in mypy_protobuf/main.py, which installs to an executable protoc-gen-mypy. On windows it installs to protoc-gen-mypy.exe

Features

See Changelog for full listing

Bring comments from .proto files to docstrings in .pyi files

Comments in the .proto files on messages, fields, enums, enum variants, extensions, services, and methods will appear as docstrings in .pyi files. Useful in IDEs for showing completions with comments.

Types enum int values more strongly

Enum int values produce stubs which wrap the int values in NewType

enum MyEnum {
  FOO = 0;
  BAR = 1;
}

Will yield an enum type wrapper whose methods type to MyEnum.V rather than int. This allows mypy to catch bugs where the wrong enum value is being used.

mypy-protobuf autogenerates an instance of the EnumTypeWrapper as follows.

class MyEnum(_MyEnum, metaclass=_MyEnumEnumTypeWrapper):
    pass
class _MyEnum:
    V = typing.NewType('V', builtins.int)
class _MyEnumEnumTypeWrapper(google.protobuf.internal.enum_type_wrapper._EnumTypeWrapper[_MyEnum.V], builtins.type):
    DESCRIPTOR: google.protobuf.descriptor.EnumDescriptor = ...
    FOO = MyEnum.V(0)
    BAR = MyEnum.V(1)
FOO = MyEnum.V(0)
BAR = MyEnum.V(1)

_MyEnumEnumTypeWrapper extends the EnumTypeWrapper to take/return MyEnum.V rather than int MyEnum is an instance of the EnumTypeWrapper.

  • Use _MyEnum and of metaclass is an implementation detail to make MyEnum.V a valid type w/o a circular dependency

Calling code may be typed as follows.

In python >= 3.7

# Need [PEP 563](https://www.python.org/dev/peps/pep-0563/) to postpone evaluation of annotations
from __future__ import annotations  # Not needed with python>=3.10
def f(x: MyEnum.V):
    print(x)
f(MyEnum.Value("FOO"))

Note that for usages of cast, the type of x must be quoted until upstream protobuf includes V

cast('MyEnum.V', x)

Supports generating type wrappers for fields and maps

M.proto

message M {
  uint32 user_id = 1 [(mypy_protobuf.casttype)="mymod.UserId"
  map<uint32, string> email_by_uid = 2 [
    (mypy_protobuf.keytype)="path/to/mymod.UserId",
    (mypy_protobuf.valuetype)="path/to/mymod.Email"
  ];
}

mymod.py

UserId = NewType("UserId", int)
Email = NewType("Email", Text)

py_generic_services

If py_generic_services is set in your proto file, then mypy-protobuf will generate service stubs. If you want GRPC stubs instead - use the GRPC instructions.

readable_stubs

If readable_stubs is set, mypy-protobuf will generate easier-to-read stubs. The downside to this approach - is that it's possible to generate stubs which do not pass mypy - particularly in the case of name collisions. mypy-protobuf defaults to generating stubs with fully qualified imports and mangled global-level identifiers to defend against name collisions between global identifiers and field names.

If you're ok with this risk, try it out!

protoc --python_out=output/location --mypy_out=readable_stubs:output/location

relax_strict_optional_primitives

If you are using proto3, then primitives cannot be represented as NULL on the wire - only as their zero value. By default mypy-protobuf types message constructors to have non-nullable primitives (eg int instead of Optional[int]). python-protobuf itself will internally convert None -> zero value. If you intentionally want to use this behavior, set this flag! We recommend avoiding this, as it can lead to developer error - confusing NULL and 0 as distinct on the wire. However, it may be helpful when migrating existing proto2 code, where the distinction is meaningful

protoc --python_out=output/location --mypy_out=relax_strict_optional_primitives:output/location

Output suppression

To suppress output, you can run

protoc --python_out=output/location --mypy_out=quiet:output/location

GRPC

This plugin provides stubs generation for grpcio generated code.

protoc \
    --python_out=output/location \
    --mypy_out=output/location \
    --grpc_out=output/location \
    --mypy_grpc_out=output/location

Note that generated code for grpc will work only together with code for python and locations should be the same. If you need stubs for grpc internal code we suggest using this package https://github.com/shabbyrobe/grpc-stubs

Targeting python2 support

mypy-protobuf's drops support for targeting python2 with version 3.0. If you still need python2 support -

python3 -m pip install mypy_protobuf==2.10
protoc --python_out=output/location --mypy_out=output/location
mypy --target-version=2.7 {files}

Contributing

Contributions to the implementation are welcome. Please run tests using ./run_test.sh. Ensure code is formatted using black.

pip3 install black
black .

Contributors

Dropboxers

Others

Licence etc.

  1. License: Apache 2.0.
  2. Copyright attribution: Copyright (c) 2017 Dropbox, Inc.
  3. External contributions to the project should be subject to Dropbox's Contributor License Agreement (CLA): https://opensource.dropbox.com/cla/

Rate & Review

Great Documentation0
Easy to Use0
Performant0
Highly Customizable0
Bleeding Edge0
Responsive Maintainers0
Poor Documentation0
Hard to Use0
Slow0
Buggy0
Abandoned0
Unwelcoming Community0
100