await file_obj.credential.close() : TypeError: object NoneType can't be used in 'await' expression #431
Comments
|
fsspec / adlfs use an async internally. can you try using the credentials from `azure.identity.aio` instead?
… On Oct 6, 2023, at 10:43 AM, ELToulemonde ***@***.***> wrote:
My problem
At the end of my python script, I get a clean up error : TypeError: object NoneType can't be used in 'await' expression on
Traceback (most recent call last):
File ".../lib/python3.10/weakref.py", line 667, in _exitfunc
f()
File ".../lib/python3.10/weakref.py", line 591, in __call__
return info.func(*info.args, **(info.kwargs or {}))
File ".../lib/python3.10/site-packages/fsspec/asyn.py", line 103, in sync
raise return_result
File ".../lib/python3.10/site-packages/fsspec/asyn.py", line 56, in _runner
result[0] = await coro
File ".../lib/python3.10/site-packages/adlfs/utils.py", line 78, in close_credential
await file_obj.credential.close()
TypeError: object NoneType can't be used in 'await' expression
Which is weird because, I don't do anything asynchronious.
A reproducible example
At least as much as I can:
from azure.identity import ChainedTokenCredential, ManagedIdentityCredential, AzureCliCredential
azure_cli = AzureCliCredential()
managed_identity = ManagedIdentityCredential()
CREDENTIAL_CHAIN = ChainedTokenCredential(managed_identity, azure_cli)
import pandas as pd
***@***.***/path_to_parquets.parquet", storage_options={"credential": credential_chain})
print("Done")
I do get the "Done" printed before failure.
My config
ubuntu,
python 3.10,
azure-storage-blob==12.16.0
pandas==2.0.0
pyarrow==11.0.0
adlfs==2023.9.0
—
Reply to this email directly, view it on GitHub <#431> or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKAOIVBR7SYSQMVOV7SKMTX6ARJZBFKMF2HI4TJMJ2XIZLTSOBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJLJONZXKZNENZQW2ZNLORUHEZLBMRPXI6LQMWBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTLDTOVRGUZLDORPXI6LQMWSUS43TOVS2M5DPOBUWG44SQKSHI6LQMWVHEZLQN5ZWS5DPOJ42K5TBNR2WLKJRGA4TIMBRHE4TTAVEOR4XAZNFNFZXG5LFUV3GC3DVMWVDCOJTGA2DOMBSGA2KO5DSNFTWOZLSUZRXEZLBORSQ>.
You are receiving this email because you are subscribed to this thread.
Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
|
@TomAugspurger @ELToulemonde I get the same error, no difference importing |
|
Same error here, any solutions? |
|
Importing Python 3.10.13 on Ubuntu. |
|
I am aware that using |
|
fsspec uses asyncio internally. I'd recommend people use credentials from |
|
That may solve one of our issue: I am using adlfs as part of a complex code that use ThreadPool. At the end of the run, I get this message that do not change the exit code, so not fatal but looks a bit ugly: I did not manage to create a small reproducable example ... |
Reproduction exampleJust create an from adlfs import AzureBlobFileSystem
from azure.identity import DefaultAzureCredential
AzureBlobFileSystem(
account_name="_",
credential=DefaultAzureCredential(),
)When the code exits an exception is raised: Traceback (most recent call last):
File "/usr/lib/python3.12/weakref.py", line 666, in _exitfunc
f()
File "/usr/lib/python3.12/weakref.py", line 590, in __call__
return info.func(*info.args, **(info.kwargs or {}))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".venv/lib/python3.12/site-packages/fsspec/asyn.py", line 103, in sync
raise return_result
File ".venv/lib/python3.12/site-packages/fsspec/asyn.py", line 56, in _runner
result[0] = await coro
^^^^^^^^^^
File ".venv/lib/python3.12/site-packages/adlfs/utils.py", line 78, in close_credential
await file_obj.credential.close()
TypeError: object NoneType can't be used in 'await' expressionWhat about
|
The import io
import os
import adlfs
from azure.identity.aio import DefaultAzureCredential
credential = DefaultAzureCredential()
azfs = adlfs.AzureBlobFileSystem(
account_name=<storage-account-name>,
credential=credential
)
with io.BytesIO() as buf:
buf.write(str('hello from the byterealm!').encode())
buf.seek(0)
azfs.write_bytes('/container/path/msg.txt', buf)
file_output = azfs.read_text('/container/path/msg.txt')
print(file_output)Giving the print out: $ python remote-storage.py
hello from the byterealm!
Using an asynchronous object in a threadpool sounds like a headache... you could create a |
|
I had the same issue but I needed to use the exclude_interactive_browser_credential=False parameter which is not available in the azure.identity.aio one. What I ended up is creating a custom class from the sync (not the aio), adding a dumb close async function so fsspec is happy: class AsyncDefaultAzureCredentialWrapper(AsyncTokenCredential):
"""
This class wraps the synchronous DefaultAzureCredential to provide an asynchronous interface.
It allows the use of DefaultAzureCredential with fsspec, which requires asynchronous credentials.
The close method is overridden to be a no-op to prevent issues when fsspec tries to close the credential.
"""
def __init__(self, **kwargs):
# Initialize the synchronous DefaultAzureCredential
self._credential = DefaultAzureCredential(**kwargs)
async def get_token(self, *scopes, **kwargs):
#Asynchronously get a token for the specified scopes: this method runs the synchronous get_token method in an executor to avoid blocking.
loop = asyncio.get_event_loop()
token = await loop.run_in_executor(None, self._credential.get_token, *scopes, **kwargs)
return AccessToken(token.token, token.expires_on)
async def close(self):
#No-op close method to prevent issues when fsspec tries to close the credential.
pass
# Create the async credential from the sync one, with the no-op close
credential = AsyncDefaultAzureCredentialWrapper(exclude_interactive_browser_credential=False) |
My problem
At the end of my python script, I get a clean up error :
TypeError: object NoneType can't be used in 'await' expressionComplete trace is :
Which is weird because, I don't do anything asynchronious.
A reproducible example
At least as much as I can:
I do get the "Done" printed before failure.
My config
The text was updated successfully, but these errors were encountered: