Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for external annotations in the typing module #600

Closed
till-varoquaux opened this issue Dec 13, 2018 · 37 comments
Closed

Add support for external annotations in the typing module #600

till-varoquaux opened this issue Dec 13, 2018 · 37 comments

Comments

@till-varoquaux
Copy link
Contributor

till-varoquaux commented Dec 13, 2018

We propose adding an Annotated type to the typing module to decorate existing types with context-specific metadata. Specifically, a type T can be annotated with metadata x via the typehint Annotated[T, x]. This metadata can be used for either static analysis or at runtime. If a library (or tool) encounters a typehint Annotated[T, x] and has no special logic for metadata x, it should ignore it and simply treat the type as T. Unlike the no_type_check functionality that current exists in the typing module which completely disables typechecking annotations on a function or a class, the Annotated type allows for both static typechecking of T (e.g., via MyPy or Pyre, which can safely ignore x) together with runtime access to x within a specific application. We believe that the introduction of this type would address a diverse set of use cases of interest to the broader Python community.

Motivating examples:

READING binary data

The struct module provides a way to read and write C structs directly from their byte representation. It currently relies on a string representation of the C type to read in values:

record = b'raymond   \x32\x12\x08\x01\x08'
name, serialnum, school, gradelevel = unpack('<10sHHb', record)

The documentation suggests using a named tuple to unpack the values and make this a bit more tractable:

from collections import namedtuple
Student = namedtuple('Student', 'name serialnum school gradelevel')
Student._make(unpack('<10sHHb', record))
# Student(name=b'raymond   ', serialnum=4658, school=264, gradelevel=8)

However, this recommendation is somewhat problematic; as we add more fields, it's going to get increasingly tedious to match the properties in the named tuple with the arguments in unpack.

Instead, annotations can provide better interoperability with a type checker or an IDE without adding any special logic outside of the struct module:

from typing import NamedTuple
UnsignedShort = Annotated[int, struct.ctype('H')]
SignedChar = Annotated[int, struct.ctype('b')]

@struct.packed
class Student(NamedTuple):
  # MyPy typechecks 'name' field as 'str' 
  name: Annotated[str, struct.ctype("<10s")]
  serialnum: UnsignedShort
  school: SignedChar
  gradelevel: SignedChar

# 'unpack' only uses the metadata within the type annotations
Student.unpack(record))
# Student(name=b'raymond   ', serialnum=4658, school=264, gradelevel=8)

dataclasses

Here's an example with dataclasses that is a problematic from the typechecking standpoint:

from dataclasses import dataclass, field

@dataclass
class C:
  myint: int = 0
  # the field tells the @dataclass decorator that the default action in the
  # constructor of this class is to set "self.mylist = list()"
  mylist: List[int] = field(default_factory=list)

Even though one might expect that mylist is a class attribute accessible via C.mylist (like C.myint is) due to the assignment syntax, that is not the case. Instead, the @dataclass decorator strips out the assignment to this attribute, leading to an AttributeError upon access:

C.myint  # Ok: 0
C.mylist  # AttributeError: type object 'C' has no attribute 'mylist'

This can lead to confusion for newcomers to the library who may not expect this behavior. Furthermore, the typechecker needs to understand the semantics of dataclasses and know to not treat the above example as an assignment operation in (which translates to additional complexity).

It makes more sense to move the information contained in field to an annotation:

@dataclass
class C:
    myint: int = 0
    mylist: Annotated[List[int], field(default_factory=list)]

# now, the AttributeError is more intuitive because there is no assignment operator
C.mylist  # AttributeError

# the constructor knows how to use the annotations to set the 'mylist' attribute 
c = C()
c.mylist  # []

The main benefit of writing annotations like this is that it provides a way for clients to gracefully degrade when they don't know what to do with the extra annotations (by just ignoring them). If you used a typechecker that didn't have any special handling for dataclasses and the field annotation, you would still be able to run checks as though the type were simply:

class C:
    myint: int = 0
    mylist: List[int]

lowering barriers to developing new types

Typically when adding a new type, we need to upstream that type to the typing module and change MyPy, PyCharm, Pyre, pytype, etc. This is particularly important when working on open-source code that makes use of our new types, seeing as the code would not be immediately transportable to other developers' tools without additional logic (this is a limitation of MyPy plugins, which allow for extending MyPy but would require a consumer of new typehints to be using MyPy and have the same plugin installed). As a result, there is a high cost to developing and trying out new types in a codebase. Ideally, we should be able to introduce new types in a manner that allows for graceful degradation when clients do not have a custom MyPy plugin, which would lower the barrier to development and ensure some degree of backward compatibility.

For example, suppose that we wanted to add support for tagged unions to Python. One way to accomplish would be to annotate TypedDict in Python such that only one field is allowed to be set:

Currency = Annotated(
  TypedDict('Currency', {'dollars': float, 'pounds': float}, total=False),
  TaggedUnion,
)

This is a somewhat cumbersome syntax but it allows us to iterate on this proof-of-concept and have people with non-patched IDEs work in a codebase with tagged unions. We could easily test this proposal and iron out the kinks before trying to upstream tagged union to typing, MyPy, etc. Moreover, tools that do not have support for parsing the TaggedUnion annotation would still be able able to treat Currency as a TypedDict, which is still a close approximation (slightly less strict).

Details of proposed changes to typing

Syntax

Annotated is parameterized with a type and an arbitrary list of Python values that represent the annotations. Here are the specific details of the syntax:

  • The first argument to Annotated must be a valid typing type
  • Multiple type annotations are supported (Annotated supports variadic arguments): Annotated[int, ValueRange(3, 10), ctype("char")]
  • When called with no extra arguments Annotated returns the underlying value: Annotated[int] == int
  • The order of the annotations is preserved and matters for equality checks: Annotated[int, ValueRange(3, 10), ctype("char")] != Annotated[int, ctype("char"), ValueRange(3, 10)]
  • Nested Annotated types are flattened, with metadata ordered starting with the innermost annotation: Annotated[Annotated[int, ValueRange(3, 10)], ctype("char")] == Annotated[int, ValueRange(3, 10), ctype("char")]``
  • Duplicated annotations are not removed: Annotated[int, ValueRange(3, 10)] != Annotated[int, ValueRange(3, 10), ValueRange(3, 10)]

consuming annotations

Ultimately, the responsibility of how to interpret the annotations (if at all) is the responsibility of the tool or library encountering the Annotated type. A tool or library encountering an Annotated type can scan through the annotations to determine if they are of interest (e.g., using isinstance).

Unknown annotations:
When a tool or a library does not support annotations or encounters an unknown annotation it should just ignore it and treat annotated type as the underlying type. For example, if we were to add an annotation that is not an instance of struct.ctype to the annotation for name (e.g., Annotated[str, 'foo', struct.ctype("<10s")]), the unpack method should ignore it.

Namespacing annotations:
We do not need namespaces for annotations since the class used by the annotations acts as a namespace.

Multiple annotations:
It's up to the tool consuming the annotations to decide whether the client is allowed to have several annotations on one type and how to merge those annotations.

Since the Annotated type allows you to put several annotations of the same (or different) type(s) on any node, the tools or libraries consuming those annotations are in charge of dealing with potential duplicates. For example, if you are doing value range analysis you might allow this:

T1 = Annotated[int, ValueRange(-10, 5)]
T2 = Annotated[T1, ValueRange(-20, 3)]

Flattening nested annotations, this translates to:

T2 = Annotated[int, ValueRange(-10, 5), ValueRange(-20, 3)]

An application consuming this type might choose to reduce these annotations via an intersection of the ranges, in which case T2 would be treated equivalently to Annotated[int, ValueRange(-10, 3)].

An alternative application might reduce these via a union, in which case T2 would be treated equivalently to Annotated[int, ValueRange(-20, 5)].

In this example whether we reduce those annotations using union or intersection can be context dependant (covarient vs contravariant); this is why we have to preserve all of them and let the consumers decide how to merge them.

Other applications may decide to not support multiple annotations and throw an exception.

related bugs

  • issues 482: Mixing typing and non-typing information in annotations has some discussion about this problem but none of the proposed solutions (using intersection types, passing dictionaries of annotations) seemed to garner enough steam. We hope this solution is non-intrusive and compelling enough to make it in the standard library.
@ilevkivskyi
Copy link
Member

I like this proposal. It will allow easy experimentation with type system features. I have two questions:

  • Why do we need to allow the single-argument form Annotated[int]?
  • Should we allow subscripting annotated types, for example currently one can write Vec = List[Tuple[T, T]] (where T is a type variable), and then Vec[int] will be equivalent to List[Tuple[int, int]]. Should we allow (and how) Vec = Annotated[List[Tuple[T, T]], MaxLen(10)]; v: Vec[int]?

@ilevkivskyi
Copy link
Member

Also an organizational question: do we need a (mini-)PEP for this? I would say yes (this post can be a starting point for a draft). @gvanrossum what do you think?

@till-varoquaux
Copy link
Contributor Author

@ilevkivskyi
_ We don't really need the single-argument form, I was just trying to be consistent with Union
_ Yes, supporting generic aliases would be great and would allow us to give more of a first class feeling to some of the Annotated type when we're experimenting.

I'm happy to turn this into a PEP if you think that this is the way forward.

@ilevkivskyi
Copy link
Member

We don't really need the single-argument form

OK, let's remove it.

I'm happy to turn this into a PEP if you think that this is the way forward.

I think this is the best way forward (in particular because it aims at improving cross-typechecker compatibility). If you will start working on it, then I would recommend skipping the dataclass example, it doesn't look very convincing TBH.

@valtron
Copy link

valtron commented Jan 6, 2019

If you haven't already considered this, I think Annotated[None, ...] should be allowed for cases where you want the type to be inferred, but still want to add metadata.

Another use case for this: Annotated[T, ClassVar, Final]; the current approach (e.g. ClassVar[T]) is kind of hacky since ClassVar/Final aren't really types, just specially-cased versions of Annotated.

@ilevkivskyi
Copy link
Member

None may be a valid type, so I would rather propose Annotated[..., Immutable] with literal ..., see also #276.

@pfalcon
Copy link

pfalcon commented Jan 9, 2019

I was referred here from #604. Indeed, this would cover a usecase there too. But for me this looks a bit too verbose, the same concern as for communicated in #482 .

I like the idea of "PEP" in the sense of "more thorough document", though I'm not sure if it would be Python-as-a-language wide PEP, or just local PEP for "typing" project/module.

In either case, considering and discussing different alternatives is a must for PEP, and I'd like a list of other options considered and comments re: them.

Here's my 2 cents:

  1. Why instead Annotated[Type, Ann1, Ann2, ...] not have (Type, Ann1, Ann2, ...) ? By unbreaking grammar a bit, this would allow to annotate entire functions by putting just comma-separate annotations in return annotation position, e.g.:
def foo() -> NoReturn, NoRaise:  # Perhaps an infinite loop?

For function params/variables, parens would be required, but it's still less visual noise than Annotated, which is also a pretty long word, which will easily cause need for line wrapping when annotating e.g. a class method with more than a couple params.

  1. I find it interesting that in the description above, a typo with parens instead of square brackets is made:
Currency = Annotated(
  TypedDict('Currency', {'dollars': float, 'pounds': float}, total=False),
  TaggedUnion,
)

So, given that Annotated isn't really a typing annotation itself, but a kind of meta-annotation, perhaps blindly following typing types syntax isn't a requirement, and other alternatives can be considered. I'm not sure about implementation complications, but imagine that by overriding __new__ and/or __call__ it's doable. Using call syntax would allow to use keyword arguments for example.

Let me clarify again that the above is just to list possible alternative solutions. I'm clearly in favor of p.1 ;-). And while using keyword args is always cute, this statement from the original description:

We do not need namespaces for annotations since the class used by the annotations acts as a namespace.

Indeed, we can't namespace keywords. So, effectively we're trading:

Annotate(T, very_readable_but_not_namespaced_keyword="10s")

with

Annotate[T, my_module.VeryReadableAnnotationName("10s")]

So, choosing between original proposal and p.2, I'm in favor of the original proposal: namespaces are important to avoid incompatibilities and mess.

But even more so I'm in favor of p.1, and would be keen to hear criticism of it.

@ilevkivskyi
Copy link
Member

Why instead Annotated[Type, Ann1, Ann2, ...] not have (Type, Ann1, Ann2, ...) ?

There are several reasons that I see:

  • Annotated types can appear in a nested position, that can cause confusions: Callable[[A, B], C] is too similar to Callable[[(A, B)], C], but the meaning would be very different.
  • With the (...) syntax users cannot create generic aliases, for example Vec = Annotated[List[Tuple[T, T]], MaxLen(10)]; Vec[int] will work, but the same will not work with (...) syntax.
  • This will be simpler for people who use annotations for runtime purposes (they will not need to expect a tuple in any position, we can use the same API as for other typing constructs).

@pfalcon
Copy link

pfalcon commented Jan 10, 2019

There are several reasons that I see:
...

I see, makes sense. The only thing then is to see if someone would be able to come up with something shorter than Annotated. Though fairly speaking one can just do A = Annotated or whatever, so that's covered too.

@ilevkivskyi
Copy link
Member

Though fairly speaking one can just do A = Annotated or whatever, [...]

Exactly.

@jstasiak
Copy link
Contributor

jstasiak commented Mar 5, 2019

I also rather like this proposal. I don't think a relevant PEP has been created yet?

One thing worth considering here is how Annotated would interact with typing.get_type_hints().

Having

@struct.packed
class Student(NamedTuple):
  name: Annotated[str, struct.ctype("<10s")]

I see three ways of approaching this:

  1. get_type_hints() returns exactly what's stored in the annotations, so:
get_type_hints(Student) == {'name': Annotated[str, struct.ctype("<10s")]}
  1. get_type_hints() returns only types and then:
get_type_hints(Student) == {'name': str}
  1. A flag is introduced to control what's get_type_hints() gonna return, for example:
get_type_hints(Student) == {'name': str}  # probably
get_type_hints(Student, what=ONLY_TYPES) == {'name': str}
get_type_hints(Student, what=ONLY_EXTRA) == {'name': [struct.ctype("<10s")]}
get_type_hints(Student, what=ALL) == {'name': Annotated[str, struct.ctype("<10s")]}

I'm a fan of the second variant, it'd require a new function or functions to be introduced so that client code can get the relevant subset of the annotations.

@ilevkivskyi
Copy link
Member

The PEP draft has been posted on Python-ideas a while ago, see https://mail.python.org/pipermail/python-ideas/2019-January/054908.html

IIUC the current process is that you need to find a core dev sponsor for the PEP (I can be one, or maybe @gvanrossum?) and then make a PR to the python/peps repo taking into account the comments the comments that appeared on Python-ideas.

Re get_type_hints(), I am in favor of the third variant with a boolean flag (likeinclude_extras) that will be False by default to maintain current behavior.

@till-varoquaux
Copy link
Contributor Author

till-varoquaux commented Apr 3, 2019

@ilevkivskyi: Thanks for your involvement, I was a bit slammed with work but I am finally getting back to this.
We'll turn this into a proper PEP real soon and I really appreciate your feedback. We'd love to have onboard either as a sponsor or a co-author if you're interested. The points I see that currently need addressing are:

Come up with a compelling example of using Annotated as an alias.

As noted the syntax is pretty cumbersome unless we use aliases. e.g.:

T = typevar('T')
Opaque = Annotated[T, MyLib.Opaque()]

@dataclass
class V:
   name: str
   uid: Opaque[bytes]

I am not sure that this example is a compelling enough one.

Remove the dataclass example.

I thought it was compelling but I'm obviously not objective...

get_type

How would get_type_hints(Student, what=ONLY_EXTRA) work with nested annotations?

@struct.packed
class Student(NamedTuple):
  names: List[Annotated[str, struct.ctype("<10s")]]

I guess that means that we should go with include_extras (or no args at all)?

@jstasiak
Copy link
Contributor

jstasiak commented Apr 3, 2019

Come up with a compelling example of using Annotated as an alias

I'd like to provide an example for this (disclaimer: I'm not objective here either, as I co-maintain the library in question): Injector (a dependency injection framework) has a way to mark constructor parameters are noninjectable (so that the framework won't attempt to provide a value and it's obvious to the reader /documentation value/). Currently the way it's done is:

    # ...
    @noninjectable('user_id')
    def __init__(self, service: Service, user_id: int):
        # ...

With Annotated and generic alias support we could do this:

# Injector code
T = TypeVar('T')
Noninjectable = Annotated[T, some_marker_which_does_not_matter_here]


# client code
    # ...
    def __init__(self, service: Service, user_id: Noninjectable[int]):
        # ...

@jstasiak
Copy link
Contributor

jstasiak commented Apr 5, 2019

Another example I thought about just now: a mutability/immutability marker so that external tools could statically verify that code is not mutating something it shouldn't (pure theoretical thing though).

# ok
def fun1(data: Dict[str, int]) -> None:
    data['asd'] = 1

# error
def fun2(data: Immutable[Dict[str, int]]) -> None:
    data['asd'] = 1

@till-varoquaux
Copy link
Contributor Author

@jstasiak I really like that second example. The first example was great but required a bit more context.

@ilevkivskyi
Copy link
Member

@till-varoquaux

We'll turn this into a proper PEP real soon and I really appreciate your feedback. We'd love to have onboard either as a sponsor or a co-author if you're interested.

Thanks! I will be glad to be a sponsor of this PEP.

Come up with a compelling example of using Annotated as an alias.

Your Opaque example is OK, also Immutable[...] by @jstasiak looks great.

I guess that means that we should go with include_extras (or no args at all)?

TBH I would prefer include_extras, we can reconsider this depending on the feedback we get after publishing the PEP.

@ilevkivskyi
Copy link
Member

@till-varoquaux I think the next step is making a PR with an updated PEP text to the python/peps repo, you can probably grab the next available number 592.

@till-varoquaux
Copy link
Contributor Author

python/peps#1014

@brugz
Copy link

brugz commented Jul 22, 2019

Hi folks, first comment in an OSS community like this, so here goes:

Is there maybe a documentation piece here too that we can think about?

I see folks using type annotations in conjunction with simple one-line comments to produce a sortof-pseudo docstring to describe the nature of the annotation.

Perhaps Annotated[T, x] could also take d, where d is a docstring like object describing the Annotation:

That could look something like this (borrowed example from jstasiak@):

# Injector code
T = TypeVar('T')
Noninjectable = Annotated[
    T,
    some_marker_which_does_not_matter_here,
    'A noninjectable type used for ...']


# client code
    # ...
    def __init__(self, service: Service, user_id: Noninjectable[int]):
        # ...

We could have it populate something similar to __doc__ and that could be consistent with PEP 257 in a way:

https://www.python.org/dev/peps/pep-0257/

https://docs.python.org/3/reference/datamodel.html

Anyways, first time commentor wdyt?

@ilevkivskyi
Copy link
Member

@brugz
Annotated takes arbitrary number of arguments Annotated[T, x, y, z], so this pattern is already possible to use. I don't think there is any need for standardisation here, since a marker unrecognised by a type-checker should be ignored by the current spec, this includes any strings that appear at some positions.

@brugz
Copy link

brugz commented Aug 7, 2019

@ilevkivskyi

I hope I'm not detracting too much from the main thread, but here are some of my thoughts:

I had a quick look at Annotated, and its base classes. I don't see __doc__ being implemented though (it might be by way of __getattr__ but I didn't bother to dig too far)

I also see support for better annotations around Exception handling, which I'm also generally supportive of, because I've already been bitten by 'surprise' exceptions which aren't annotated anywhere a few times:
#604

I see folks declaring 'type aliases' at the module level but they're very static, and often ugly. Sometimes they take the form Optional[List[..., ...]] and then some big ugly data-structures... Annotated can address some of this, and also the Exceptions concern.

Annotated is interesting, because it can help to define really rich type annotations, including sanity-checking behaviors, example-data, and documentation, etc.

@Ayplow
Copy link

Ayplow commented Aug 17, 2019

Is there any good use case for the Annotated type ignoring TypeVars in the metadata? Something like

Doc = Annotated[T1, constants.DOC, T2]
@documented
def fn(arg: Doc[cls, "docstring"]):
  ...

is much nicer syntax than the alternative

def fn(arg: Annotated[cls, constants.DOC, "docstring"]):
  ...

Seems like it should be fine for it to only support top level typevars if its a problem

@ilevkivskyi
Copy link
Member

@TomSputz there is even simpler way:

Doc = Annotated

def fn(arg: Doc[int, "Some info here"]) -> None:
    ...

Because by agreement type checkers must ignore any annotations they don't understand.

@Ayplow
Copy link

Ayplow commented Aug 21, 2019

The problem is that at runtime there's no difference between Annotated[int, "Some info here"] and your Doc[int, "Some info here"], so documentation tools consuming this would have to be static. For example, this would preclude any modification to __doc__, and therefore the help command. Meaning that anyone wanting to fully document their function would have to repeat themselves in their docstring

@till-varoquaux

This comment has been minimized.

@valtron
Copy link

valtron commented Aug 21, 2019

@Ayplow Annotated[..., Doc("docstring")]?

@till-varoquaux
Copy link
Contributor Author

re-posting because my previous reply wasn't formatted properly

There are a couple of corner cases that make handling TypeVars in the annotations themselves a bit tricky.

  1. Passing values to annotations: "Some info here" is not a type and we probably should not accept filling TypeVars with values that aren't types. We could leverage the new Literal type:
Doc = Annotated[T1, constants.DOC, T2]

def fn(arg: Doc[int, Literal["Some info here"]]) -> None:
    ...

This seems somewhat kludgy and will place strict restrictions on the types of the literals you can pass as arguments (Literal doesn't support tuples...).

  1. Substituting TypeVars nested in annotations. Ideally arguments to Annotated would be self-contained; it's confusing to have constants.DOC apply to the argument directly following it. We now need to know the arity of the arguments to Annotated if we want to tell them apart. You'd probably want something more along those lines:

    Doc = Annotated[T1, annotations.Doc[T2]]

    It's not entirely clear how we'd let get_type_hints() perform TypeVar subsitutions. PEP-593 was written in part to support non type checking use cases (database query mapping, RPC parameter marshalling) so we wouldn't want to force all the arguments to Annotated to be valid types.

    I see a couple of scenarios:

    • get_type_hints() only performs TypeVar substitution on annotations that are classes inheriting from Generic.

    • We go all out and add LiteralVar and GenericValue to enable complex substitutions.

      • LiteralVars enable us to accepts literals as arguments. We can restrict the type of the literals we accept by passing a type LiteralVar 's constructor.
      • GenericValue takes a callable and a list of arguments (which can be instances of LiteralVar or GenericValue). We expand GenericValue[fn, args...] by
        • replacing any TypeVar or LiteralVar that appear in args...
        • calling fn on the expanded arguments

      E.g.:

      DOCSTRING = LiteralVar("DOCSTRING", str)
      Doc = Annotated[T, GenericValue[annotations.Doc, DOCSTRING]]
      
      def fn(arg: Doc[int, "Some info here"]) -> None:
        ...

      Doc[int, "Sone info here"] would get expanded to Doc[int, Annotations.Doc("Some info here")].

As we expand typing to be more flexible, we're slowly backing ourselves into template meta-programming... Meta-programming is a rabbit hole we do not want to explore within the scope of this PEP.

I'd love to see python support constructs such as c++11's constexpr * as it would address a lot of the issues we have with aliases, variable substitutions and literals. This would warrant a whole new PEP and require significant changes in the language itself.

@valtron
Copy link

valtron commented Sep 29, 2019

Another use case for this PR in mypy 0.730:

You can ignore only errors with specific error codes on a particular line by using a # type: ignore[code, ...] comment.

... which would be Annotated[..., Ignore(code)] here.

jstasiak added a commit to python-injector/injector that referenced this issue Oct 16, 2019
There's an implementation of PEP 593 draft in typing_extensions and mypy
supports it already.

See also:

* typing module discussion (pre-PEP): python/typing#600
* python-ideas discussion (pre-PEP): https://mail.python.org/pipermail/python-ideas/2019-January/054908.html
* The actual PEP: https://www.python.org/dev/peps/pep-0593/
* typing-sig PEP discussion (ongoing): https://mail.python.org/archives/list/[email protected]/thread/CZ7N3M3PGKHUY63RWWSPTICVOAVYI73D/
jstasiak added a commit to python-injector/injector that referenced this issue Oct 16, 2019
There's an implementation of PEP 593 draft in typing_extensions and mypy
supports it already.

See also:

* typing module discussion (pre-PEP): python/typing#600
* python-ideas discussion (pre-PEP): https://mail.python.org/pipermail/python-ideas/2019-January/054908.html
* The actual PEP: https://www.python.org/dev/peps/pep-0593/
* typing-sig PEP discussion (ongoing): https://mail.python.org/archives/list/[email protected]/thread/CZ7N3M3PGKHUY63RWWSPTICVOAVYI73D/
jstasiak added a commit to python-injector/injector that referenced this issue Oct 16, 2019
There's an implementation of PEP 593 draft in typing_extensions and mypy
supports it already.

See also:

* typing module discussion (pre-PEP): python/typing#600
* python-ideas discussion (pre-PEP): https://mail.python.org/pipermail/python-ideas/2019-January/054908.html
* The actual PEP: https://www.python.org/dev/peps/pep-0593/
* typing-sig PEP discussion (ongoing): https://mail.python.org/archives/list/[email protected]/thread/CZ7N3M3PGKHUY63RWWSPTICVOAVYI73D/
jstasiak added a commit to python-injector/injector that referenced this issue Oct 16, 2019
There's an implementation of PEP 593 draft in typing_extensions and mypy
supports it already.

See also:

* typing module discussion (pre-PEP): python/typing#600
* python-ideas discussion (pre-PEP): https://mail.python.org/pipermail/python-ideas/2019-January/054908.html
* The actual PEP: https://www.python.org/dev/peps/pep-0593/
* typing-sig PEP discussion (ongoing): https://mail.python.org/archives/list/[email protected]/thread/CZ7N3M3PGKHUY63RWWSPTICVOAVYI73D/
@QuentinSoubeyran
Copy link

QuentinSoubeyran commented Jul 14, 2020

What is the proper way to access extra annotations at runtime ? The typing.get_type_hints(include_extras=True) function just gives back the result of Annotated[T, x] and I couldn't find a way to get the annotations -- in either the PEP or the upcoming documentation.

From the source code, annotations are stored in the __metadata__ attribute, but since this is a dunder attribute, it shouldn't be accessed according to this documentation

Perhaps __metadata__ should be added to the doc ?

EDIT: Added the link to the doc essentially reserving all dunder names in any context

@jstasiak
Copy link
Contributor

jstasiak commented Jul 14, 2020 via email

@QuentinSoubeyran
Copy link

I'd like to get all the x in Annotated[T, x, ...] at runtime to test if a particular one is present. After re-reading the doc and checking the code, typing.get_args() is exactly what I want -- sorry for missing it and bothering everyone !

@mahi1073
Copy link

mahi1073 commented Nov 5, 2020

Nice& well keep it up

@Conchylicultor
Copy link

What is the proper way to check whether a typing annotation is a Annotated ?

annotation = Annotated[int, 123]

# The naive way does not work
isinstance(annotation, Annotated)

# Works but rely on implementation details
isinstance(annotation, typing._AnnotatedAlias)

# Works but feel fragile
hasattr(annotation, '__metadata__')

get_type_hints(fn, include_extras=True) returns annotated but does not say anything of whether a specific item is Annotated or not. Also the Annotation could be nested inside Dict[str, List[Annotated[...]]]. Similarly get_args(Annotated[int, int]) and get_args(Tuple[int, int]) will return the same value, so it can't be used to check whether an annotation is Annotated.

@QuentinSoubeyran
Copy link

@Conchylicultor You can use typing.get_origin() (after retrieving type hints with typing.get_type_hints(obj, include_extras=True)). Here is also how to retrieve the type and annotations from the annotated type hint:

from typing import Annotated, get_origin, get_args
MyType = Annotated[int, 123, "metadata"]

assert get_origin(MyType) is Annotated
assert get_args(MyType)[0] == int
assert get_args(MyType)[1:] == (123, "metadata")

@Conchylicultor
Copy link

@QuentinSoubeyran it seems to work. Thanks. I guess I got confused because Annotated[int, ...].__origin__ is int while get_origin(Annotated[int, ...]) is Annotated

@JelleZijlstra
Copy link
Member

Closing this since the PEP has been accepted and implemented.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests