-
Notifications
You must be signed in to change notification settings - Fork 242
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for external annotations in the typing module #600
Comments
I like this proposal. It will allow easy experimentation with type system features. I have two questions:
|
Also an organizational question: do we need a (mini-)PEP for this? I would say yes (this post can be a starting point for a draft). @gvanrossum what do you think? |
@ilevkivskyi I'm happy to turn this into a PEP if you think that this is the way forward. |
OK, let's remove it.
I think this is the best way forward (in particular because it aims at improving cross-typechecker compatibility). If you will start working on it, then I would recommend skipping the dataclass example, it doesn't look very convincing TBH. |
If you haven't already considered this, I think Another use case for this: |
|
I was referred here from #604. Indeed, this would cover a usecase there too. But for me this looks a bit too verbose, the same concern as for communicated in #482 . I like the idea of "PEP" in the sense of "more thorough document", though I'm not sure if it would be Python-as-a-language wide PEP, or just local PEP for "typing" project/module. In either case, considering and discussing different alternatives is a must for PEP, and I'd like a list of other options considered and comments re: them. Here's my 2 cents:
For function params/variables, parens would be required, but it's still less visual noise than
So, given that Let me clarify again that the above is just to list possible alternative solutions. I'm clearly in favor of p.1 ;-). And while using keyword args is always cute, this statement from the original description:
Indeed, we can't namespace keywords. So, effectively we're trading:
with
So, choosing between original proposal and p.2, I'm in favor of the original proposal: namespaces are important to avoid incompatibilities and mess. But even more so I'm in favor of p.1, and would be keen to hear criticism of it. |
There are several reasons that I see:
|
I see, makes sense. The only thing then is to see if someone would be able to come up with something shorter than |
Exactly. |
I also rather like this proposal. I don't think a relevant PEP has been created yet? One thing worth considering here is how Having @struct.packed
class Student(NamedTuple):
name: Annotated[str, struct.ctype("<10s")] I see three ways of approaching this:
get_type_hints(Student) == {'name': Annotated[str, struct.ctype("<10s")]}
get_type_hints(Student) == {'name': str}
get_type_hints(Student) == {'name': str} # probably
get_type_hints(Student, what=ONLY_TYPES) == {'name': str}
get_type_hints(Student, what=ONLY_EXTRA) == {'name': [struct.ctype("<10s")]}
get_type_hints(Student, what=ALL) == {'name': Annotated[str, struct.ctype("<10s")]} I'm a fan of the second variant, it'd require a new function or functions to be introduced so that client code can get the relevant subset of the annotations. |
The PEP draft has been posted on Python-ideas a while ago, see https://mail.python.org/pipermail/python-ideas/2019-January/054908.html IIUC the current process is that you need to find a core dev sponsor for the PEP (I can be one, or maybe @gvanrossum?) and then make a PR to the python/peps repo taking into account the comments the comments that appeared on Python-ideas. Re |
@ilevkivskyi: Thanks for your involvement, I was a bit slammed with work but I am finally getting back to this. Come up with a compelling example of using
|
I'd like to provide an example for this (disclaimer: I'm not objective here either, as I co-maintain the library in question): Injector (a dependency injection framework) has a way to mark constructor parameters are noninjectable (so that the framework won't attempt to provide a value and it's obvious to the reader /documentation value/). Currently the way it's done is: # ...
@noninjectable('user_id')
def __init__(self, service: Service, user_id: int):
# ... With Annotated and generic alias support we could do this: # Injector code
T = TypeVar('T')
Noninjectable = Annotated[T, some_marker_which_does_not_matter_here]
# client code
# ...
def __init__(self, service: Service, user_id: Noninjectable[int]):
# ... |
Another example I thought about just now: a mutability/immutability marker so that external tools could statically verify that code is not mutating something it shouldn't (pure theoretical thing though). # ok
def fun1(data: Dict[str, int]) -> None:
data['asd'] = 1
# error
def fun2(data: Immutable[Dict[str, int]]) -> None:
data['asd'] = 1 |
@jstasiak I really like that second example. The first example was great but required a bit more context. |
Thanks! I will be glad to be a sponsor of this PEP.
Your
TBH I would prefer |
@till-varoquaux I think the next step is making a PR with an updated PEP text to the python/peps repo, you can probably grab the next available number 592. |
Hi folks, first comment in an OSS community like this, so here goes: Is there maybe a documentation piece here too that we can think about? I see folks using type annotations in conjunction with simple one-line comments to produce a sortof-pseudo docstring to describe the nature of the annotation. Perhaps That could look something like this (borrowed example from jstasiak@):
We could have it populate something similar to https://www.python.org/dev/peps/pep-0257/ https://docs.python.org/3/reference/datamodel.html Anyways, first time commentor wdyt? |
@brugz |
I hope I'm not detracting too much from the main thread, but here are some of my thoughts: I had a quick look at I also see support for better annotations around Exception handling, which I'm also generally supportive of, because I've already been bitten by 'surprise' exceptions which aren't annotated anywhere a few times: I see folks declaring 'type aliases' at the module level but they're very static, and often ugly. Sometimes they take the form Optional[List[..., ...]] and then some big ugly data-structures...
|
Is there any good use case for the Doc = Annotated[T1, constants.DOC, T2]
@documented
def fn(arg: Doc[cls, "docstring"]):
... is much nicer syntax than the alternative def fn(arg: Annotated[cls, constants.DOC, "docstring"]):
... Seems like it should be fine for it to only support top level typevars if its a problem |
@TomSputz there is even simpler way: Doc = Annotated
def fn(arg: Doc[int, "Some info here"]) -> None:
... Because by agreement type checkers must ignore any annotations they don't understand. |
The problem is that at runtime there's no difference between |
This comment has been minimized.
This comment has been minimized.
@Ayplow |
re-posting because my previous reply wasn't formatted properly There are a couple of corner cases that make handling
Doc = Annotated[T1, constants.DOC, T2]
def fn(arg: Doc[int, Literal["Some info here"]]) -> None:
... This seems somewhat kludgy and will place strict restrictions on the types of the literals you can pass as arguments (
As we expand I'd love to see python support constructs such as c++11's |
Another use case for this PR in mypy 0.730:
... which would be |
There's an implementation of PEP 593 draft in typing_extensions and mypy supports it already. See also: * typing module discussion (pre-PEP): python/typing#600 * python-ideas discussion (pre-PEP): https://mail.python.org/pipermail/python-ideas/2019-January/054908.html * The actual PEP: https://www.python.org/dev/peps/pep-0593/ * typing-sig PEP discussion (ongoing): https://mail.python.org/archives/list/[email protected]/thread/CZ7N3M3PGKHUY63RWWSPTICVOAVYI73D/
There's an implementation of PEP 593 draft in typing_extensions and mypy supports it already. See also: * typing module discussion (pre-PEP): python/typing#600 * python-ideas discussion (pre-PEP): https://mail.python.org/pipermail/python-ideas/2019-January/054908.html * The actual PEP: https://www.python.org/dev/peps/pep-0593/ * typing-sig PEP discussion (ongoing): https://mail.python.org/archives/list/[email protected]/thread/CZ7N3M3PGKHUY63RWWSPTICVOAVYI73D/
There's an implementation of PEP 593 draft in typing_extensions and mypy supports it already. See also: * typing module discussion (pre-PEP): python/typing#600 * python-ideas discussion (pre-PEP): https://mail.python.org/pipermail/python-ideas/2019-January/054908.html * The actual PEP: https://www.python.org/dev/peps/pep-0593/ * typing-sig PEP discussion (ongoing): https://mail.python.org/archives/list/[email protected]/thread/CZ7N3M3PGKHUY63RWWSPTICVOAVYI73D/
There's an implementation of PEP 593 draft in typing_extensions and mypy supports it already. See also: * typing module discussion (pre-PEP): python/typing#600 * python-ideas discussion (pre-PEP): https://mail.python.org/pipermail/python-ideas/2019-January/054908.html * The actual PEP: https://www.python.org/dev/peps/pep-0593/ * typing-sig PEP discussion (ongoing): https://mail.python.org/archives/list/[email protected]/thread/CZ7N3M3PGKHUY63RWWSPTICVOAVYI73D/
What is the proper way to access extra annotations at runtime ? The From the source code, annotations are stored in the Perhaps EDIT: Added the link to the doc essentially reserving all dunder names in any context |
On 14 Jul 2020, at 20:26, QuentinSoubeyran ***@***.***> wrote:
What is the proper way to access extra annotations at runtime ? The typing.get_type_hints(include_extras=True) function just gives back the result of Annotated[T, x] and I couldn't find a way to get the annotations -- in either the PEP or the upcoming documentation.
From the source code, annotations are stored in the __metadata__ attribute, but since this is a dunder attribute, it doesn't quite look like an official API. Perhaps __metadata__ should be added to the doc ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
What’s your use case? I’m using get_type_hints(…, include_extras=True) and it’s working fine so you may want something specific here but it’s not clear what it is.
Best,
Jakub
|
I'd like to get all the |
Nice& well keep it up |
What is the proper way to check whether a typing annotation is a annotation = Annotated[int, 123]
# The naive way does not work
isinstance(annotation, Annotated)
# Works but rely on implementation details
isinstance(annotation, typing._AnnotatedAlias)
# Works but feel fragile
hasattr(annotation, '__metadata__')
|
@Conchylicultor You can use from typing import Annotated, get_origin, get_args
MyType = Annotated[int, 123, "metadata"]
assert get_origin(MyType) is Annotated
assert get_args(MyType)[0] == int
assert get_args(MyType)[1:] == (123, "metadata") |
@QuentinSoubeyran it seems to work. Thanks. I guess I got confused because |
Closing this since the PEP has been accepted and implemented. |
We propose adding an
Annotated
type to the typing module to decorate existing types with context-specific metadata. Specifically, a typeT
can be annotated with metadatax
via the typehintAnnotated[T, x]
. This metadata can be used for either static analysis or at runtime. If a library (or tool) encounters a typehintAnnotated[T, x]
and has no special logic for metadatax
, it should ignore it and simply treat the type asT
. Unlike theno_type_check
functionality that current exists in thetyping
module which completely disables typechecking annotations on a function or a class, theAnnotated
type allows for both static typechecking ofT
(e.g., via MyPy or Pyre, which can safely ignorex
) together with runtime access tox
within a specific application. We believe that the introduction of this type would address a diverse set of use cases of interest to the broader Python community.Motivating examples:
READING binary data
The
struct
module provides a way to read and write C structs directly from their byte representation. It currently relies on a string representation of the C type to read in values:The documentation suggests using a named tuple to unpack the values and make this a bit more tractable:
However, this recommendation is somewhat problematic; as we add more fields, it's going to get increasingly tedious to match the properties in the named tuple with the arguments in
unpack
.Instead, annotations can provide better interoperability with a type checker or an IDE without adding any special logic outside of the
struct
module:dataclasses
Here's an example with dataclasses that is a problematic from the typechecking standpoint:
Even though one might expect that
mylist
is a class attribute accessible viaC.mylist
(likeC.myint
is) due to the assignment syntax, that is not the case. Instead, the@dataclass
decorator strips out the assignment to this attribute, leading to anAttributeError
upon access:This can lead to confusion for newcomers to the library who may not expect this behavior. Furthermore, the typechecker needs to understand the semantics of dataclasses and know to not treat the above example as an assignment operation in (which translates to additional complexity).
It makes more sense to move the information contained in
field
to an annotation:The main benefit of writing annotations like this is that it provides a way for clients to gracefully degrade when they don't know what to do with the extra annotations (by just ignoring them). If you used a typechecker that didn't have any special handling for dataclasses and the
field
annotation, you would still be able to run checks as though the type were simply:lowering barriers to developing new types
Typically when adding a new type, we need to upstream that type to the typing module and change MyPy, PyCharm, Pyre, pytype, etc. This is particularly important when working on open-source code that makes use of our new types, seeing as the code would not be immediately transportable to other developers' tools without additional logic (this is a limitation of MyPy plugins, which allow for extending MyPy but would require a consumer of new typehints to be using MyPy and have the same plugin installed). As a result, there is a high cost to developing and trying out new types in a codebase. Ideally, we should be able to introduce new types in a manner that allows for graceful degradation when clients do not have a custom MyPy plugin, which would lower the barrier to development and ensure some degree of backward compatibility.
For example, suppose that we wanted to add support for tagged unions to Python. One way to accomplish would be to annotate TypedDict in Python such that only one field is allowed to be set:
This is a somewhat cumbersome syntax but it allows us to iterate on this proof-of-concept and have people with non-patched IDEs work in a codebase with tagged unions. We could easily test this proposal and iron out the kinks before trying to upstream tagged union to
typing
, MyPy, etc. Moreover, tools that do not have support for parsing theTaggedUnion
annotation would still be able able to treatCurrency
as aTypedDict
, which is still a close approximation (slightly less strict).Details of proposed changes to
typing
Syntax
Annotated
is parameterized with a type and an arbitrary list of Python values that represent the annotations. Here are the specific details of the syntax:Annotated
must be a validtyping
typeAnnotated[int, ValueRange(3, 10), ctype("char")]
Annotated
returns the underlying value:Annotated[int] == int
Annotated[int, ValueRange(3, 10), ctype("char")] != Annotated[int, ctype("char"), ValueRange(3, 10)]
Annotated
types are flattened, with metadata ordered starting with the innermost annotation:Annotated[Annotated[int, ValueRange(3, 10)], ctype("char")] ==
Annotated[int, ValueRange(3, 10), ctype("char")]``Annotated[int, ValueRange(3, 10)] != Annotated[int, ValueRange(3, 10), ValueRange(3, 10)]
consuming annotations
Ultimately, the responsibility of how to interpret the annotations (if at all) is the responsibility of the tool or library encountering the
Annotated
type. A tool or library encountering anAnnotated
type can scan through the annotations to determine if they are of interest (e.g., usingisinstance
).Unknown annotations:
When a tool or a library does not support annotations or encounters an unknown annotation it should just ignore it and treat annotated type as the underlying type. For example, if we were to add an annotation that is not an instance of
struct.ctype
to the annotation for name (e.g.,Annotated[str, 'foo', struct.ctype("<10s")]
), the unpack method should ignore it.Namespacing annotations:
We do not need namespaces for annotations since the class used by the annotations acts as a namespace.
Multiple annotations:
It's up to the tool consuming the annotations to decide whether the client is allowed to have several annotations on one type and how to merge those annotations.
Since the
Annotated
type allows you to put several annotations of the same (or different) type(s) on any node, the tools or libraries consuming those annotations are in charge of dealing with potential duplicates. For example, if you are doing value range analysis you might allow this:Flattening nested annotations, this translates to:
An application consuming this type might choose to reduce these annotations via an intersection of the ranges, in which case
T2
would be treated equivalently toAnnotated[int, ValueRange(-10, 3)]
.An alternative application might reduce these via a union, in which case
T2
would be treated equivalently toAnnotated[int, ValueRange(-20, 5)]
.In this example whether we reduce those annotations using union or intersection can be context dependant (covarient vs contravariant); this is why we have to preserve all of them and let the consumers decide how to merge them.
Other applications may decide to not support multiple annotations and throw an exception.
related bugs
The text was updated successfully, but these errors were encountered: