Skip to content
This repository has been archived by the owner on Feb 21, 2023. It is now read-only.

Aws redis 'Connection closed by server' exception occurs in my program running in aws ecs fargate #1360

Open
1 task done
bjjoy opened this issue May 11, 2022 · 0 comments
Labels

Comments

@bjjoy
Copy link

bjjoy commented May 11, 2022

Describe the bug

aioredis is used in my fastapi web program connected with aws redis. Running for one day or a moment, this exception
'2022-05-04 21:13:47 raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
2022-05-04 21:13:47aioredis.exceptions.ConnectionError: Connection closed by server.' occurs.

I have to restart my program running in aws ecs fargate container.

my aioredis client creation code is following
client = aioredis.from_url(settings.redis_uri, decode_responses=True)

And I am trying to re-connection after catching exception 'aioredis.exceptions.ConnectionError', it is also not usable.

To Reproduce

  1. create aws redis
  2. fastapi web app used by redis deploy to aws ecs fargate
  3. running a api set or get something in redis
  4. sometimes, this problem will occur

Expected behavior

How to solve this problem?
Maybe dynamic ip of aws redis leads to this problem, but aws redis service give a redis dns like 'www.myredis.test.com:6379'.

Logs/tracebacks

`2022-05-04 21:13:47ERROR: Exception in ASGI application
2022-05-04 21:13:47Traceback (most recent call last):
2022-05-04 21:13:47File "/usr/local/lib/python3.8/site-packages/aioredis/connection.py", line 1422, in get_connection
2022-05-04 21:13:47if await connection.can_read():
2022-05-04 21:13:47File "/usr/local/lib/python3.8/site-packages/aioredis/connection.py", line 893, in can_read
2022-05-04 21:13:47return await self._parser.can_read(timeout)
2022-05-04 21:13:47File "/usr/local/lib/python3.8/site-packages/aioredis/connection.py", line 479, in can_read
2022-05-04 21:13:47raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
2022-05-04 21:13:47aioredis.exceptions.ConnectionError: Connection closed by server.
2022-05-04 21:13:47During handling of the above exception, another exception occurred:
2022-05-04 21:13:47Traceback (most recent call last):
2022-05-04 21:13:47File "/usr/local/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 396, in run_asgi
2022-05-04 21:13:47result = await app(self.scope, self.receive, self.send)
2022-05-04 21:13:47File "/usr/local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
2022-05-04 21:13:47return await self.app(scope, receive, send)
2022-05-04 21:13:47File "/usr/local/lib/python3.8/site-packages/fastapi/applications.py", line 199, in __call__
2022-05-04 21:13:47await super().__call__(scope, receive, send)
2022-05-04 21:13:47File "/usr/local/lib/python3.8/site-packages/starlette/applications.py", line 111, in __call__
2022-05-04 21:13:47await self.middleware_stack(scope, receive, send)
2022-05-04 21:13:47File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
2022-05-04 21:13:47await self.app(scope, receive, _send)
2022-05-04 21:13:47File "/usr/local/lib/python3.8/site-packages/starlette/middleware/cors.py", line 86, in __call__
2022-05-04 21:13:47await self.simple_response(scope, receive, send, request_headers=headers)
2022-05-04 21:13:47File "/usr/local/lib/python3.8/site-packages/starlette/middleware/cors.py", line 142, in simple_response
2022-05-04 21:13:47await self.app(scope, receive, send)
2022-05-04 21:13:47File "/usr/local/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
2022-05-04 21:13:47await self.app(scope, receive, sender)
2022-05-04 21:13:47File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 566, in __call__
2022-05-04 21:13:47await route.handle(scope, receive, send)
2022-05-04 21:13:47File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 227, in handle
2022-05-04 21:13:47await self.app(scope, receive, send)
2022-05-04 21:13:47File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 41, in app
2022-05-04 21:13:47response = await func(request)
2022-05-04 21:13:47File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 201, in app
2022-05-04 21:13:47raw_response = await run_endpoint_function(
2022-05-04 21:13:47File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 148, in run_endpoint_function
2022-05-04 21:13:47return await dependant.call(**values)
2022-05-04 21:13:47File "/web/./api/monitor_api.py", line 157, in monitor_login
2022-05-04 21:13:47token = await monitor_service.monitor_login(request.app.state.redis, monitor_login_in.monitor_name, monitor_login_in.password)
2022-05-04 21:13:47File "/web/./service/monitor_service.py", line 193, in monitor_login
2022-05-04 21:13:47await redis_client.set(monitor_key, json.dumps(monitor_redis.dict()))
2022-05-04 21:13:47File "/usr/local/lib/python3.8/site-packages/aioredis/client.py", line 1082, in execute_command
2022-05-04 21:13:47conn = self.connection or await pool.get_connection(command_name, **options)
2022-05-04 21:13:47File "/usr/local/lib/python3.8/site-packages/aioredis/connection.py", line 1425, in get_connection
2022-05-04 21:13:47await connection.disconnect()
2022-05-04 21:13:47File "/usr/local/lib/python3.8/site-packages/aioredis/connection.py", line 806, in disconnect
2022-05-04 21:13:47await self._writer.wait_closed() # type: ignore[union-attr]
2022-05-04 21:13:47File "/usr/local/lib/python3.8/asyncio/streams.py", line 359, in wait_closed
2022-05-04 21:13:47await self._protocol._get_close_waiter(self)
2022-05-04 21:13:47asyncio.exceptions.CancelledError
2022-05-04 21:13:47INFO: 172.31.11.76:37616 - "POST /monitor-service/api/monitor/login HTTP/1.1" 500 Internal Server Error`

Python Version

$ python --version
Python 3.8.2

aioredis Version

$ python -m pip show aioredis

Name: aioredis
Version: 2.0.1
Summary: asyncio (PEP 3156) Redis support
Home-page: https://github.com/aio-libs/aioredis-py
Author: 
Author-email: 
License: MIT
Location: /opt/anaconda3/lib/python3.8/site-packages
Requires: async-timeout, typing-extensions
Required-by:

Additional context

No response

Code of Conduct

  • I agree to follow the aio-libs Code of Conduct
@bjjoy bjjoy added the bug label May 11, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

1 participant