Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added extras execute_batch for executemany #632

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

vir-mir
Copy link
Member

@vir-mir vir-mir commented Dec 5, 2019

I added this function, since it allows you to quickly insert values ​​into the database.

This solves the executemany problem, more details can be found in this issues psycopg/psycopg2#491

I wrote several of my performance tests that showed the following result.
insert 10000 query in database

test first https://github.com/vir-mir/kdr2019/tree/master/test2
aiopg not execute_batch: 18.12 sec
asyncpg: 10.09 sec

test second https://github.com/vir-mir/kdr2019/tree/master/test3
aiopg execute_batch: 0.64 sec
asyncpg: 10.09 sec

@codecov
Copy link

codecov bot commented Dec 5, 2019

Codecov Report

Merging #632 into master will increase coverage by 0.03%.
The diff coverage is 94.44%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #632      +/-   ##
==========================================
+ Coverage   92.18%   92.22%   +0.03%     
==========================================
  Files          10       11       +1     
  Lines        1037     1055      +18     
  Branches      121      126       +5     
==========================================
+ Hits          956      973      +17     
  Misses         65       65              
- Partials       16       17       +1
Impacted Files Coverage Δ
aiopg/extras.py 94.44% <94.44%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2911c10...cb55c94. Read the comment docs.

@lgtm-com
Copy link

lgtm-com bot commented Dec 6, 2019

This pull request introduces 1 alert when merging ccd5a46 into 2911c10 - view on LGTM.com

new alerts:

  • 1 for Unguarded next in generator

@iAnanich
Copy link

iAnanich commented Jan 2, 2020

When this could be merged? Can't wait to use it )

@@ -0,0 +1,43 @@
def _paginate(seq, page_size):

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

import itertools 


def _paginate(seq, page_size):
    it = iter(seq)
    page = list(itertools.islice(it, 0, page_size))
    while page:
        yield page
        page = list(itertools.islice(it, 0, page_size))

I think it is better solution

Copy link

@andrey-berenda andrey-berenda Jan 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with this solution you can use iterators in _paginate
for example
_paginate(range(1, 8), 3)

cur.close()


def test__paginate():

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pytest.mark.parametrize(
    'data, expected_pages',
    (
            (
                    (1, 2, 3, 4, 5, 6, 7), [
                        [1, 2, 3],
                        [4, 5, 6],
                        [7],
                    ]
            ),
            (
                    (1, 2, 3, 4, 5, 6), [
                        [1, 2, 3],
                        [4, 5, 6],
                    ]
            ),
    )
)
def test__paginate(data, expected_pages):
    for page, expected_page in itertools.zip_longest(
            _paginate(data, page_size=3),
            expected_pages,
    ):
        assert page == expected_page

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

@aio-libs aio-libs deleted a comment from CLAassistant Nov 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants