Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Heterogeneous Speculative Decoding (CPU + GPU) #5065

Closed
wants to merge 79 commits into from
Closed

Heterogeneous Speculative Decoding (CPU + GPU) #5065

wants to merge 79 commits into from

Commits on May 16, 2024

  1. hete spec decode engine

    @jiqing-feng
    jiqing-feng committed May 16, 2024
    Configuration menu
    Copy the full SHA
    aaece57 View commit details
    Browse the repository at this point in the history

Commits on May 24, 2024

  1. compile ops for cuda and cpu

    @jiqing-feng
    jiqing-feng committed May 24, 2024
    Configuration menu
    Copy the full SHA
    21fb773 View commit details
    Browse the repository at this point in the history

  2. can run hete spec decode

    @jiqing-feng
    jiqing-feng committed May 24, 2024
    Configuration menu
    Copy the full SHA
    5f02fdd View commit details
    Browse the repository at this point in the history

Commits on May 27, 2024

  1. add parameter cpu_draft_worker to run draft model on CPU

    @jiqing-feng
    jiqing-feng committed May 27, 2024
    Configuration menu
    Copy the full SHA
    8febd81 View commit details
    Browse the repository at this point in the history

  2. rm useless comments

    @jiqing-feng
    jiqing-feng committed May 27, 2024
    Configuration menu
    Copy the full SHA
    d9af7a6 View commit details
    Browse the repository at this point in the history

  3. merge main

    @jiqing-feng
    jiqing-feng committed May 27, 2024
    Configuration menu
    Copy the full SHA
    74fb5d5 View commit details
    Browse the repository at this point in the history

  4. fix conflict

    @jiqing-feng
    jiqing-feng committed May 27, 2024
    Configuration menu
    Copy the full SHA
    44acebe View commit details
    Browse the repository at this point in the history

  5. add copy comment

    @jiqing-feng
    jiqing-feng committed May 27, 2024
    Configuration menu
    Copy the full SHA
    b4b8744 View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2024

  1. rebase

    @jiqing-feng
    jiqing-feng committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    cc7998e View commit details
    Browse the repository at this point in the history

  2. fix bug

    @jiqing-feng
    jiqing-feng committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    8f7ecf3 View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2024

  1. rebase

    @jiqing-feng
    jiqing-feng committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    794613e View commit details
    Browse the repository at this point in the history

  2. fix style

    @jiqing-feng
    jiqing-feng committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    52022a5 View commit details
    Browse the repository at this point in the history

  3. rebbase

    @jiqing-feng
    jiqing-feng committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    fa40a93 View commit details
    Browse the repository at this point in the history

  4. fix style

    @jiqing-feng
    jiqing-feng committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    aa4d556 View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2024

  1. Merge branch 'main' into hete_spec_decode

    @jiqing-feng
    jiqing-feng authored Jul 4, 2024
    Configuration menu
    Copy the full SHA
    f7491eb View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2024

  1. fix format

    @jiqing-feng
    jiqing-feng committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    344f5d7 View commit details
    Browse the repository at this point in the history

  2. rebase

    @jiqing-feng
    jiqing-feng committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    53cf9b6 View commit details
    Browse the repository at this point in the history

  3. fix format

    @jiqing-feng
    jiqing-feng committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    23a4575 View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2024

  1. rebase main

    @jiqing-feng
    jiqing-feng committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    2cab72f View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2024

  1. rebase main

    @jiqing-feng
    jiqing-feng committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    185836b View commit details
    Browse the repository at this point in the history

  2. fix style

    @jiqing-feng
    jiqing-feng committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    0556d02 View commit details
    Browse the repository at this point in the history

  3. fix diff

    @jiqing-feng
    jiqing-feng committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    2eb3201 View commit details
    Browse the repository at this point in the history

  4. fix arg

    @jiqing-feng
    jiqing-feng committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    345788e View commit details
    Browse the repository at this point in the history

  5. fix match

    @jiqing-feng
    jiqing-feng committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    49d5bdf View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2024

  1. fix cmake

    @jiqing-feng
    jiqing-feng committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    12e6c5d View commit details
    Browse the repository at this point in the history

  2. fix cmake style

    @jiqing-feng
    jiqing-feng committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    e313329 View commit details
    Browse the repository at this point in the history

  3. add numa in cuda dockerfile

    @jiqing-feng
    jiqing-feng committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    32c9c9a View commit details
    Browse the repository at this point in the history

  4. use low version gcc

    @jiqing-feng
    jiqing-feng committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    e1782d3 View commit details
    Browse the repository at this point in the history

  5. rm useless link

    @jiqing-feng
    jiqing-feng committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    ff7efee View commit details
    Browse the repository at this point in the history

  6. enable TP

    @jiqing-feng
    jiqing-feng committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    1c5d8c4 View commit details
    Browse the repository at this point in the history

  7. rebase

    @jiqing-feng
    jiqing-feng committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    59de387 View commit details
    Browse the repository at this point in the history

  8. fix format

    @jiqing-feng
    jiqing-feng committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    6fc9b3b View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2024

  1. rm erro cpu cache ops

    @jiqing-feng
    jiqing-feng committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    ded4e78 View commit details
    Browse the repository at this point in the history

  2. fix cpu op import

    @jiqing-feng
    jiqing-feng committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    1eca335 View commit details
    Browse the repository at this point in the history

  3. disable cpu TP model

    @jiqing-feng
    jiqing-feng committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    b986b4d View commit details
    Browse the repository at this point in the history

  4. rebase main

    @jiqing-feng
    jiqing-feng committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    1db03a2 View commit details
    Browse the repository at this point in the history

  5. fix cpu worker core binding

    @jiqing-feng
    jiqing-feng committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    1033dcc View commit details
    Browse the repository at this point in the history

  6. fix import cpu ops

    @jiqing-feng
    jiqing-feng committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    a0f172c View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2024

  1. enable cpu TP

    @jiqing-feng
    jiqing-feng committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    14df487 View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2024

  1. add cpu-draft-worker parameter

    @jiqing-feng
    jiqing-feng committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    7bbd35b View commit details
    Browse the repository at this point in the history

  2. fix style

    @jiqing-feng
    jiqing-feng committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    c895b50 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into hete_spec_decode

    @jiqing-feng
    jiqing-feng committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    91499e1 View commit details
    Browse the repository at this point in the history

  4. fix param name

    @jiqing-feng
    jiqing-feng committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    d7b742c View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. fix cpu-draft-args

    @jiqing-feng
    jiqing-feng committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    e016db9 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into hete_spec_decode

    @jiqing-feng
    jiqing-feng authored Sep 10, 2024
    Configuration menu
    Copy the full SHA
    0d58142 View commit details
    Browse the repository at this point in the history

  3. fix cmake list to avoid amd error

    @jiqing-feng
    jiqing-feng committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    679664b View commit details
    Browse the repository at this point in the history

  4. fix ops name

    @jiqing-feng
    jiqing-feng committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    729483e View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2024

  1. fix distributed tests and disable distributed verified if cpu-draft-m… 

    …odel
    @jiqing-feng
    jiqing-feng committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    13e5e2a View commit details
    Browse the repository at this point in the history

  2. fix tests

    @jiqing-feng
    jiqing-feng committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    581c529 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. rebase

    @jiqing-feng
    jiqing-feng committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    753e1d0 View commit details
    Browse the repository at this point in the history

  2. skip build cpu if rocm and fix code style

    @jiqing-feng
    jiqing-feng committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    c670338 View commit details
    Browse the repository at this point in the history

  3. install onednn

    @jiqing-feng
    jiqing-feng committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    c3e9488 View commit details
    Browse the repository at this point in the history

  4. fix install onednn position

    @jiqing-feng
    jiqing-feng committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    5d7233f View commit details
    Browse the repository at this point in the history

  5. ondnn install

    @jiqing-feng
    jiqing-feng committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    8bfc4e6 View commit details
    Browse the repository at this point in the history

  6. fix cpu op

    @jiqing-feng
    jiqing-feng committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    e01732e View commit details
    Browse the repository at this point in the history

  7. change dockerfile base image to ubuntu22.04

    @jiqing-feng
    jiqing-feng committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    07eb1a1 View commit details
    Browse the repository at this point in the history

  8. fix cmake list

    @jiqing-feng
    jiqing-feng committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    27da2ee View commit details
    Browse the repository at this point in the history

  9. install libc6

    @jiqing-feng
    jiqing-feng committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    da1728a View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. revert dockerfile to ubuntu 20.04

    @jiqing-feng
    jiqing-feng committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    a883fce View commit details
    Browse the repository at this point in the history

  2. disable avx512 to pass cpu compile

    @jiqing-feng
    jiqing-feng committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    6aba90b View commit details
    Browse the repository at this point in the history

  3. reuse multi step worker for CPU

    @jiqing-feng
    jiqing-feng committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    83bc114 View commit details
    Browse the repository at this point in the history

  4. fix SDPA assert

    @jiqing-feng
    jiqing-feng committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    d057f34 View commit details
    Browse the repository at this point in the history

  5. fix format

    @jiqing-feng
    jiqing-feng committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    77e97e2 View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. Merge branch 'main' into hete_spec_decode

    @jiqing-feng
    jiqing-feng authored Sep 23, 2024
    Configuration menu
    Copy the full SHA
    f7a2585 View commit details
    Browse the repository at this point in the history

  2. fix cpu build

    @jiqing-feng
    jiqing-feng committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    f4f6987 View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2024

  1. fix log

    @jiqing-feng
    jiqing-feng committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    58e333f View commit details
    Browse the repository at this point in the history

  2. rebase

    @jiqing-feng
    jiqing-feng committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    9cfa493 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2024

  1. Merge branch 'main' into hete_spec_decode

    @jiqing-feng
    jiqing-feng committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    15677b9 View commit details
    Browse the repository at this point in the history

  2. fix setup and spec_decode tests

    @jiqing-feng
    jiqing-feng committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    758d0bc View commit details
    Browse the repository at this point in the history

  3. add cpu+gpu spec decode tests

    @jiqing-feng
    jiqing-feng committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    a16d40a View commit details
    Browse the repository at this point in the history

  4. fix ruff

    @jiqing-feng
    jiqing-feng committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    90083c3 View commit details
    Browse the repository at this point in the history

  5. fix hip setup

    @jiqing-feng
    jiqing-feng committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    0554443 View commit details
    Browse the repository at this point in the history

  6. disable seed in test speculative decoding on cpu + gpu cause it will … 

    …confuse the generators device
    @jiqing-feng
    jiqing-feng committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    17898ef View commit details
    Browse the repository at this point in the history

  7. enable seed tests

    @jiqing-feng
    jiqing-feng committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    d0232ef View commit details
    Browse the repository at this point in the history

  8. Merge branch 'main' into hete_spec_decode

    @jiqing-feng
    jiqing-feng committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    bc7878f View commit details
    Browse the repository at this point in the history

  9. fix quant ops

    @jiqing-feng
    jiqing-feng committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    22ba6df View commit details
    Browse the repository at this point in the history

  10. add cutlass_scaled_mm cpu ops

    @jiqing-feng
    jiqing-feng committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    4323ca3 View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2024

  1. static_scaled_int8_quant cpu ops

    @jiqing-feng
    jiqing-feng committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    7721026 View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2024

  1. Merge branch 'main' into hete_spec_decode

    @jiqing-feng
    jiqing-feng authored Oct 10, 2024
    Configuration menu
    Copy the full SHA
    c8e7314 View commit details
    Browse the repository at this point in the history