Build dependency Intel Data Movers Library #100
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Run tests on CI Run long tests (navy/bench) every day on CI Run CI on prebuild docker image Run only centos build on CI Update docker file used in CI Centos8 is EOL Disable failing clang-format-check Add extra param to build-package.sh Add scripts for rebuilding/pushing docker images Taken from: pmem/dev-utils-kit@30794c3 Extend CI to rebuild docker automatically Update build-cachelib-docker.yml Do not use shallow clone to make sure Docker rebuild logic works correctly. Added required packages to install Intel ittapi Update CI to use intel/CacheLib repo (pmem#17) Add multi-tier navy benchmark and run it on CI - fix navy multi-tier config for NUMA bindings added code coverage support in CacheLib Adding libdml to CentOS docker image (pmem#53) only exclude allocator-test-NavySetupTestm, shm-test-test_page_size tests added perf and numactl to docker packages --------------------------------------------- one large commit for all CI and code coverage see above for the change history.
to utilize combined locking.
This includes printing: - allocSize - allocated memory size - memory usage fraction
…art 2) fix for compressed ptr (upstream) -> compress from false to true
for different pool sizes. We also use getPoolSize(pid), to get total size from all pools across allocators. It also fixes the tiering sizes (pulls changes from what was issue75 rebased commit that did not make it into upstream commits). Rebased to use ramCacheSize.
for the compressed ptr changes that were introduced upstream. - Includes later cosmetic changes added by sounak 9cb5c29
fix for rolling stats (on multi-tier to be followed by multi-tier rolling stats implementation in the following commit)
Hot queue iterator for 2Q. Will start at Hot queue and move to Warm queue if hot queue is exhausted. Useful for promotion semantics if using 2Q replacement. rebased on to develop and added some tests.
- transparent item movement - multi-tier combined locking with exclusive bit (pmem#38) with refactored incRef to support returning the result of markMoving (fail if already moving or exclusvie bit is set) option. - add tests (updated for numa bindings - post combined locking) for transparent item movement
-updated configs for numa bindings
This would lead to deadlock (.e.g in forEachChainedItem) if the child is moving (e.g. marked by Slab Release thread). Instead treat moving bit only to prevent freeing the item and do all synchronization on parent.
Background data movement using periodic workers. Attempts to evict/promote items per given thresholds for each class. These reduce p99 latency since there is a higher chance that an allocation slot is free in the tier we are allocating in. fix race in promotion where releaseBackToAllocator was being called before wakeUpWaiters. reinsert to mm container on failed promotion
…move to fail - updated slab release logic for move failure, but there is still an issue with slab movement. currently investigating.
The assumption for moving items was that once item is unmarked no one can add new waiters for that item. However, since incrementing item ref count was not done under the MoveMap lock, there was a race: item could have been unmarked right after incRef returned incFailedMoving.
* Fix issue with token creation * Do not increment evictFail* stats if evictFailConcurrentFill were incremented
updated the docker gcc version to 12 --------- Co-authored-by: Matt Rae <[email protected]>
- we first check if an item is expired under mmContainer lock and if so mark it for eviction so it is recycled back up to allocateInternalTier.
instead of always inserting to topmost tier
* Chained item movement between tiers - currently sync on the parent item for moving. - updated tests accordingly, note that we can no longer swap parent item if chained item is being moved for slab release. * added some debug checks around chained item check * fix slab release behavior if no movecb
Track latency of per item eviction/promotion between memory tiers
* Set dependencies to working versions and use dependencies from build context, instead of downloading cachelib:develop during build step. This makes sure that dependencies are always build in proper versions. * Fix CacheStats size
byrnedj
force-pushed
the
develop
branch
2 times, most recently
from
09d7bab to
ebfca17
Compare
byrnedj
force-pushed
the
develop
branch
4 times, most recently
from
9ba4e79 to
c1ff397
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is meant to add Intel DML as an external package in CacheLib. DML is needed for Intel DSA-based background data movement between memory tiers in CacheLib.
This change is