Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve pit performance #1673

Open
wants to merge 36 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
192ddc8
improve pit performance
Oct 19, 2023
afff257
improve pit cache
Oct 20, 2023
6c214aa
lint
Oct 20, 2023
a144bc9
deal with empty data
Oct 20, 2023
3ed3f17
add pit backend: FilePITStorage
Oct 26, 2023
61c31ca
improve docstring
Oct 26, 2023
d82ab8d
remove index file check
Oct 26, 2023
8702049
pit rewrite does not need index
Oct 26, 2023
e07487d
fix typo
Oct 31, 2023
8d96bd6
make sure dir exist
Oct 31, 2023
4213b68
fix parents not exist
Oct 31, 2023
8a354ef
fix pitstorage update
Oct 31, 2023
dbfe153
check dtype
Nov 1, 2023
20889ca
fix empty data
Nov 1, 2023
5c16123
lint
PaleNeutron Nov 2, 2023
31c3747
deal with empty data file
Nov 7, 2023
ef9242e
remove useless function
Nov 28, 2023
87d65e1
improve pit performance
Oct 19, 2023
b53bae6
improve pit cache
Oct 20, 2023
23f16b9
lint
Oct 20, 2023
1a349d0
deal with empty data
Oct 20, 2023
f340776
add pit backend: FilePITStorage
Oct 26, 2023
38a04b6
improve docstring
Oct 26, 2023
07cff6b
remove index file check
Oct 26, 2023
bdf8060
pit rewrite does not need index
Oct 26, 2023
e3fff65
fix typo
Oct 31, 2023
c754290
make sure dir exist
Oct 31, 2023
74fd9cb
fix parents not exist
Oct 31, 2023
41648b9
fix pitstorage update
Oct 31, 2023
ca0d4bb
check dtype
Nov 1, 2023
de9e6cf
fix empty data
Nov 1, 2023
e093a83
lint
PaleNeutron Nov 2, 2023
52c5cba
deal with empty data file
Nov 7, 2023
8dfc393
remove useless function
Nov 28, 2023
e42496a
Merge branch 'pit_fix' of https://github.com/PaleNeutron/qlib into pi…
Sep 3, 2024
958291e
Merge remote-tracking branch 'upstream/main' into pit_fix
Sep 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
pit rewrite does not need index
  • Loading branch information
John Lyu committed Jan 12, 2024
commit bdf80605a6872865f4a40701d2e5a4bfb315c231
4 changes: 2 additions & 2 deletions qlib/data/storage/storage.py
Original file line number Diff line number Diff line change
@@ -574,7 +574,7 @@ def write(self, data_array: np.ndarray, index: int = None):
"""
raise NotImplementedError("Subclass of FeatureStorage must implement `write` method")

def rewrite(self, data: Union[List, np.ndarray, Tuple], index: int):
def rewrite(self, data: Union[List, np.ndarray, Tuple]):
"""overwrite all data in FeatureStorage with data

Parameters
@@ -585,7 +585,7 @@ def rewrite(self, data: Union[List, np.ndarray, Tuple], index: int):
data start index
"""
self.clear()
self.write(data, index)
self.write(data, 0)

def update(self, data_array: np.ndarray) -> None:
"""update data to storage, replace current data from start_date to end_date with given data_array