-
Notifications
You must be signed in to change notification settings - Fork 158
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
added extras execute_batch for executemany #632
base: master
Are you sure you want to change the base?
Conversation
Codecov Report
@@ Coverage Diff @@
## master #632 +/- ##
==========================================
+ Coverage 92.18% 92.22% +0.03%
==========================================
Files 10 11 +1
Lines 1037 1055 +18
Branches 121 126 +5
==========================================
+ Hits 956 973 +17
Misses 65 65
- Partials 16 17 +1
Continue to review full report at Codecov.
|
This pull request introduces 1 alert when merging ccd5a46 into 2911c10 - view on LGTM.com new alerts:
|
ccd5a46
to
cb55c94
Compare
When this could be merged? Can't wait to use it ) |
@@ -0,0 +1,43 @@ | |||
def _paginate(seq, page_size): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
import itertools
def _paginate(seq, page_size):
it = iter(seq)
page = list(itertools.islice(it, 0, page_size))
while page:
yield page
page = list(itertools.islice(it, 0, page_size))
I think it is better solution
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
with this solution you can use iterators in _paginate
for example
_paginate(range(1, 8), 3)
cur.close() | ||
|
||
|
||
def test__paginate(): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pytest.mark.parametrize(
'data, expected_pages',
(
(
(1, 2, 3, 4, 5, 6, 7), [
[1, 2, 3],
[4, 5, 6],
[7],
]
),
(
(1, 2, 3, 4, 5, 6), [
[1, 2, 3],
[4, 5, 6],
]
),
)
)
def test__paginate(data, expected_pages):
for page, expected_page in itertools.zip_longest(
_paginate(data, page_size=3),
expected_pages,
):
assert page == expected_page
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks!
I added this function, since it allows you to quickly insert values into the database.
This solves the
executemany
problem, more details can be found in this issues psycopg/psycopg2#491I wrote several of my performance tests that showed the following result.
insert 10000 query in database
test first https://github.com/vir-mir/kdr2019/tree/master/test2
aiopg not execute_batch: 18.12 sec
asyncpg: 10.09 sec
test second https://github.com/vir-mir/kdr2019/tree/master/test3
aiopg execute_batch: 0.64 sec
asyncpg: 10.09 sec