Skip to content

2026-04-27-sparsity#212

Open
Slicky325 wants to merge 61 commits intoiclr-blogposts:mainfrom
Slicky325:main
Open

2026-04-27-sparsity#212
Slicky325 wants to merge 61 commits intoiclr-blogposts:mainfrom
Slicky325:main

Conversation

@Slicky325
Copy link
Contributor

@Slicky325 Slicky325 commented Jan 3, 2026

OpenReview Submission Thread

Checklist before opening a PR

  • I am opening a pull request against the main branch of the 2026 repo.

  • My post and all associated references to it are all lowercase, i.e

      2026-04-27-Sample-Test.md               -> 2026-04-27-sample-test.md
  assets/img/2026-04-27-Sample-Test/ 	-> assets/img/2026-04-27-sample-test/

  • The title of my PR is exactly the name of my markdown file

    • i.e. _posts/2026-04-27-[submission-name].md would require a PR name 2026-04-27-[submission-name]

  • I have anonymized my post: my author's list is Anonymous, and there is no potential
    content which can reveal my/my collaborators identities.

  • My post matches the formatting requirements, including (but not limited to):

    • I have ONLY MODIFIED files in the following locations (failure to do so will result in
      your PR automatically being closed!):
      • a Markdown (or HTML) file in _posts/ with the format _posts/2026-04-27-[submission-name].md (or .html)
      • static image assets added to assets/img/2026-04-27-[submission-name]/
      • interactive HTML figures added to assets/html/2026-04-27-[submission-name]/
      • citations in a bibtex file in assets/bibliography/2026-04-27-[submission-name].bib
    • I have a short 2-3 sentence abstract in the description field of my front-matter
    • I have a table of contents, formatted using the toc field of my front-matter
    • My bibliography is correctly formatted, using a .bibtex file as per the sample post

Any other comments

zer0-data and others added 30 commits December 5, 2025 18:51
@zer0-data
Create blog post on sparsity in LLMs
cd2a9f3 
Add a comprehensive blog post on sparsity in Large Language Models, discussing challenges and solutions related to self-attention mechanisms, attention sinks, and dynamic sparsity techniques.
@zer0-data
Add files via upload
3e9691f
@zer0-data
Rename assets/img/3.png to assets/img/2026-04-27-sparsity/3.png
0c01b34
@zer0-data
Add files via upload
281c4c1
@zer0-data
Create bibliography file for language model research
f142179 
Added a bibliography file containing various articles related to language models and attention mechanisms.
@zer0-data
Revise figures and captions in sparsity.md
b7d77fc 
Updated figures and captions in the sparsity article to enhance clarity and presentation.
@zer0-data
Bold figure captions for improved emphasis
da0809f
@zer0-data
Refine description and introduction for clarity
be02b8b 
Removed redundant phrasing regarding time complexity in the description and introduction sections.
@zer0-data
Clarify hybrid approach for robust sparse attention
24d3e45 
Revised the explanation of robust sparse attention methods, emphasizing the importance of retaining initial tokens and the impact on attention matrix representation.
@zer0-data
Enhance KV cache explanation with diagram and details
3fef46f 
Added a diagram explaining KV caching and its efficiency benefits in attention computation. Expanded on dynamic KV cache management and introduced the H2O method for retaining important tokens.
@zer0-data
Add files via upload
7a070fe
@tanvi102006
Replaced heatmap to be more readable
7077fec
@zer0-data
Clean up introduction section in sparsity.md
ae59690 
Removed note about table of contents usage and extra line.
@zer0-data
Update mathematical notation in sparsity.md
02e8fa9
@zer0-data
Clarify Masked Self-Attention mechanism details
008df04 
Expanded explanation of the Masked Self-Attention mechanism and its role in autoregressive tasks.
@zer0-data
Update title for sparsity blog post
b2ca50b
@zer0-data
Update title to clarify complexity reduction method
5505491
@zer0-data
Change blog post title and enhance content on sparsity
2020dbe 
Updated the title of the blog post to 'Sparsity' and refined the content to focus on the role of sparsity in improving efficiency in large language models.
@zer0-data
Update title and description in sparsity.md
e62fe80
@zer0-data
Fix title formatting in sparsity.md
ea15df0
@zer0-data
Update title to reflect complexity notation
e45a8af
@zer0-data
Update title to clarify complexity reduction
e01c26f
@zer0-data
algos
3bef0bd
@zer0-data
Enhance MInference algorithms and explanations
e34c2fa 
Refined MInference algorithms for sparse attention patterns, including dynamic execution strategies for Vertical-Slash and Block-Sparse patterns. Enhanced explanations and added figures for clarity.
@zer0-data
Refine explanations of attention patterns and sinks
352dcc3 
Revised text to clarify observations on attention mechanisms and their implications for sparse attention methods, including references to figures.
@zer0-data
Revise Softmax discussion and introduce Softpick
24ecb13 
Updated the explanation of Softmax and introduced Softpick as an alternative, detailing its benefits and addressing issues with traditional methods.
@zer0-data
Revise Softmax discussion and introduce Softpick
a22d1c5 
Updated the explanation of Softmax and introduced Softpick as a solution to its limitations.
@zer0-data
Update bibliography by removing and adding articles
0b48fe4 
Removed outdated articles and added a new entry for 'Star Attention'.
@zer0-data
Enhance attention sinks explanation with citations
4f3c41c 
Updated the discussion on attention sinks, including citations and clarifications on their implications for sparse attention and softmax alternatives.
@zer0-data
Enhance sparsity.md with citations and references
24cec4d 
Updated citations and added references to MInference, XAttention, SeerAttention, Tidal Decode, and Heavy Hitter Hypothesis in the sparsity discussion.
tanvi102006 and others added 30 commits December 7, 2025 01:58
@tanvi102006
speedup
3f567d6
@zer0-data
Enhance clarity on attention mechanisms and sparsity
76357e8 
Refactor and clarify sections on attention sinks and their implications in Transformer models. Update explanations of softmax alternatives and introduce concepts like Sliding Window Attention and Star Attention.
@zer0-data
Enhance clarity and detail in sparsity methods section
28a8468 
Refined sections on MInference, XAttention, and Tidal Decode, enhancing clarity and detail. Adjusted performance metrics and descriptions for better understanding of dynamic sparsity techniques.
@zer0-data
Refine LLMs and sparsity description for clarity
883e6e3 
Revised the description of LLMs and sparsity, removing quadratic complexity notation and improving clarity. Adjusted text for better readability and flow.
@zer0-data
Clean up author metadata in sparsity.md
06d7509 
Removed duplicate author entries from the metadata.
@zer0-data
Add captions to figures in sparsity.md
2313901 
Updated figure inclusions with captions for clarity.
@zer0-data
Correct Softpick citation in sparsity.md
6327f4d 
Updated Softpick citation for clarity and accuracy.
@zer0-data
Update 2026-04-27-sparsity.md
9c732a6
@zer0-data
Refine figure captions and clarify attention mechanisms
ee1443f 
Updated figure caption to clarify latency scaling with sequence length. Revised text for clarity and consistency in discussing attention mechanisms and attention sinks.
@zer0-data
Reorder sparsity patterns in figure description
05a9c91
@zer0-data
Fix formatting and improve clarity in sparsity.md
06c0a76
@zer0-data
Fix description of online inference strategies
c5bea69
@zer0-data
Enhance discussion on Regularity and introduce Star Attention
66dd62e 
Clarified the explanation of Regularity in attention methods and introduced Star Attention as a solution to propagation issues.
@zer0-data
Refine explanation of Regularity in SWA
14fd500 
Clarify the description of Regularity in Sliding Window Attention, changing 'unadjacent' to 'distant'.
@zer0-data
Remove captions from figure includes in sparsity.md
926f585
@zer0-data
Update OOM description in KV Cache optimization section
44d1dd1 
Clarified the description of out of memory errors (OOMs) in the context of KV cache growth.
@tanvi102006
Beyond Softmax
58e179a
@zer0-data
grammar
76a2d97
@zer0-data
Refine content and examples in sparsity.md
aca84e1 
Updated the content in the sparsity.md file to improve clarity and consistency, including adjustments to figures and examples.
@zer0-data
Update 2026-04-27-sparsity.md
faf0c6a
@zer0-data
Refine methodology section in sparsity.md
7d1a256 
Clarified the methodology steps for EpiCache, enhancing the explanation of partitioning, block-wise prefill, and decoding processes.
@zer0-data
Enhance eviction policy explanation in sparsity.md
a42bac1 
Clarified the equation for the retention strategy in the eviction policy section, explaining the role of $S_i$ and $D_i$.
@zer0-data
Revise description in sparsity.md
088b689 
Updated the description section to improve clarity and maintain consistency.
@zer0-data
Fix wording in eviction policy explanation
bdc3b81
@Slicky325
Merge pull request #1 from iclr-blogposts/main
8b0e6e6 
2026-04-27-sparsity
@zer0-data
Add section on Softpick computational overhead and stability
484d8e1 
Discuss the computational overhead and training stability issues associated with the Softpick algorithm, including its impact on gradient norms and model dynamics.
@zer0-data
Add empirical evaluation section on speed-accuracy trade-off
8028ee4 
Added a section on empirical evaluation of speed-accuracy trade-off for various attention frameworks, including a comparison table.
@zer0-data
Replace anonymous authors with specific contributors
1dde9fb 
Updated authors section with specific names and affiliations.
@zer0-data
Remove authors section from sparsity.md
bbf09c9 
Removed authors section from the sparsity post.
@zer0-data
Revise empirical evaluation of speed-accuracy trade-off
1da1066 
Updated the table format and added details for inference frameworks, including speedup, memory reduction, and accuracy trade-offs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Slicky325 @zer0-data @tanvi102006