Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Follow up] Improve performance when rendering sample waveforms #7695

Open
wants to merge 11 commits into
base: master
Choose a base branch
from

Conversation

khoidauminh
Copy link
Contributor

@khoidauminh khoidauminh commented Feb 9, 2025

This PR adds additional changes on top of #7366:

  • Attempts to resolve the garbage rendering issue when viewing large samples.
Details In the old code, `m_paintPixmap` in `SampleClipView` is allocated to the size of the whole clip, and painted at (0, 0) no matter what. On large samples, when the width exceeds 32768 (QPixmap's resolution limit), garbage data will be shown.

In this PR, m_paintPixmap is allocated to the paintEvent's size (but with the clip's height), and painted onto the clip at the paintEvent's X coordinate. To make sure the m_paintPixmap doesn't get drawn on the wrong coordinate (due to something overlaying the clip and changing the paintEvent region), we paint on the coordinate obtained from the last full redraw.

ảnh

This change allows some fields of VisualizeParameters to be omitted (drawRect).

Should resolve the sample clips part of #3378

@sakertooth
Copy link
Contributor

I also noticed there's some unfinished business I have with the draw and generation code possibly. I want to follow up with that too in this PR if that's okay with you.

@khoidauminh
Copy link
Contributor Author

khoidauminh commented Feb 10, 2025

Ye sure @sakertooth

Btw there are some changes regarding the VisualizeParameters so check that out first

@bratpeki
Copy link
Member

Glad to see this!

@khoidauminh, could you load any samples with a low pitch (kicks, toms, basses, etc) and zoom in on the waveform? I tried this in OFP and it generates a fill on the part of the waveform below 0!

Must've missed checking short samples in my testing, as I was so focused on long ones! 🤣

@khoidauminh
Copy link
Contributor Author

@bratpeki I think I know what the issue is. Originally the Peak struct min and max default values are +infinity and -infinity respectively

But in a commit it was changed to +infinity and 0 (::max() and ::min() in source code), so max gets stuck at 0 when the waveform sits below 0, making a fill

Simply change the min() back to -max() and it's all good (currently away from laptop so can't do yet)

@sakertooth
Copy link
Contributor

sakertooth commented Feb 10, 2025

@bratpeki I think I know what the issue is. Originally the Peak struct min and max default values are +infinity and -infinity respectively

But in a commit it was changed to +infinity and 0 (::max() and ::min() in source code), so max gets stuck at 0 when the waveform sits below 0, making a fill

Simply change the min() back to -max() and it's all good (currently away from laptop so can't do yet)

std::numeric_limits<float>::min() gives back 0? It shouldn't.

@sakertooth
Copy link
Contributor

We might want to use std::numeric_limits<float>::infinity() instead @khoidauminh, it should work better than ::max and ::min.

@bratpeki
Copy link
Member

After that fix has been applied, is this ready for testing? I view it as an extension of 7366, so it only makes sense to take a look at it as well.

@sakertooth
Copy link
Contributor

sakertooth commented Feb 10, 2025

Hi @bratpeki and @khoidauminh, I pushed a change that uses std::numeric_limits<float>::infinity() and -std::numeric_limits<float>::infinity() for the initial minimum and maximum values for Peak's respectively (another one incoming to simplify type conversions in the draw code). I'm considering to make other changes such as referencing the original sample data instead of unnecessarily copying all the sample data to a new Thumbnail, as well as figuring out how to parallelize the generation, which both should speed up generation time.

If you need my assistance with anything though let me know.

Feel free to test the new commits @bratpeki.
Also, thank you @khoidauminh for simplifying out one of those three rectangles in VisualizeParameters 👍

@khoidauminh
Copy link
Contributor Author

Is it just me or the MaxSampleThumbnailCacheSize being 32 might be a little too small? Will LMMS keep regenerating thumbnails over and over again if there are more than 32 samples, especially when they're big?

@bratpeki
Copy link
Member

Feel free to test the new commits @bratpeki.

Alright!

@bratpeki bratpeki self-assigned this Feb 11, 2025
@sakertooth
Copy link
Contributor

sakertooth commented Feb 11, 2025

Is it just me or the MaxSampleThumbnailCacheSize being 32 might be a little too small? Will LMMS keep regenerating thumbnails over and over again if there are more than 32 samples, especially when they're big?

You also have to consider that caches are generally small to avoid keeping unnecessary, unwanted data, keeping it "hot" basically. Too big of a size and you run the risk of increasing the memory usage, even when in some cases evicting a thumbnail to store a new fresh one is better than keeping the old one around. That being said, I'm more than happy to help come up with a better size that will better fit the average workload and system.

If we have more than 32 thumbnails, there are two things that can happen if another thumbnail is requested to be generated: one is a cache miss if the thumbnail is not cached, the other a cache hit if it is. The cache will only keep regenerating new thumbnails if every request for a new thumbnail after meeting that size limit wasn't cached.

Sidebar: We might also want to consider renaming ThumbnailCache to Thumbnails or remove the alias altogether to avoid confusion.

@sakertooth
Copy link
Contributor

Ideally, this idea will be extended to all resources, like sample resources and thumbnail resources, and maybe more (essentially anything that is costly to regenerate and store a lot of all at once is a resource).

@khoidauminh
Copy link
Contributor Author

@sakertooth Looking at the code, from my understanding, when the cache gets full, the unused thumbnails will be destroyed, but the least still used thumbnails will be detached instead. Existing clips will hold on to this detached cache until all of them are destroyed. So I'm guessing it's not as bad as I'm concerned. All good then

@sakertooth
Copy link
Contributor

@sakertooth Looking at the code, from my understanding, when the cache gets full, the unused thumbnails will be destroyed, but the least still used thumbnails will be detached instead. Existing clips will hold on to this detached cache until all of them are destroyed. So I'm guessing it's not as bad as I'm concerned. All good then

I guess it should be using an LRU eviction policy actually. I wont stop you from making changes, but there are plans to overhaul resource management in the future anyways as mentioned. Maybe since its really needed I might start work on it soon.

@bratpeki
Copy link
Member

Before I test, I'd like to know what to test for!

Attempts to resolve the garbage rendering issue when viewing large samples.

Could you elaborate on this? Are you reffering to the black space I was showing in my screenshots 7636?

@khoidauminh
Copy link
Contributor Author

khoidauminh commented Feb 13, 2025

@bratpeki Yes, on my Linux build it shows up as garbage like in #3378. If you're testing on Windows that's probably why it shows up as black. (There's a collapsed detail section in the desc on why it happens, if you haven't checked it out)

@bratpeki
Copy link
Member

Right, so I should expect the things listed below, and if they're accomplished, this has been tested?

  • Any sample, no matter the length or zoom level, now displays properly
  • The part of the waveform between -1.0 and 0.0 isn't filled anymore

@khoidauminh
Copy link
Contributor Author

khoidauminh commented Feb 14, 2025

Right, so I should expect the things listed below, and if they're accomplished, this has been tested?

* Any sample, no matter the length or zoom level, now displays properly

* The part of the waveform between `-1.0` and `0.0` isn't filled anymore

@bratpeki Yes, they have been tested. I have tried the following: zooming in, scrolling, overlaying another clip on the sample and zooming, overlaying a window and zooming, moving a sample (with and without an overlay). They should be fixed. But do test it anyway in order to catch more potential hidden bugs.

There's a behavior when the viewportRect doesn't contain the start of the clip (when it's outside of view or overlayed), the border will draw anyway and the sample filename will move around it until the clip start appears again.

Screencast_20250214_071652.mp4

In the view you'll also see that the sample gets shifted by the border size. I will need to do something about it.

EDIT: Ok I have no idea what happened but when I reopened lmms to work on the issue, it's gone (?) lol what Oh no it's still there, but with a trigger condition

@sakertooth
Copy link
Contributor

Hello, a user was facing an issue with a "blurry" waveform. I believe this is the antialiasing I added to make the waveform look smoother.

In this follow up, we might consider either removing it, or making it an option you can toggle.

I did add the antialiasing on a whim I suppose. I thought it looked better, especially when zooming in the AFP, but we might make it a toggle. How do you all feel about the antialiasing?

@khoidauminh
Copy link
Contributor Author

How do you all feel about the antialiasing?

I personally don't have a problem with it. But since the thumbnail we select to render is always larger than the clip size, I'm sure we won't run into a "blocky" appearance issue when we turn it off. So it's safe to remove it

@bratpeki
Copy link
Member

I, too, am pretty indifferent, at least for now. I'd have to load in a lot of samples at different zoom levels to check which works best, so in the meantime it might be best to make it a toggleable option.

@bratpeki
Copy link
Member

Right, so I should expect the things listed below, and if they're accomplished, this has been tested?

* Any sample, no matter the length or zoom level, now displays properly

* The part of the waveform between `-1.0` and `0.0` isn't filled anymore

@bratpeki Yes, they have been tested. I have tried the following: zooming in, scrolling, overlaying another clip on the sample and zooming, overlaying a window and zooming, moving a sample (with and without an overlay). They should be fixed. But do test it anyway in order to catch more potential hidden bugs.

There's a behavior when the viewportRect doesn't contain the start of the clip (when it's outside of view or overlayed), the border will draw anyway and the sample filename will move around it until the clip start appears again.

Screencast_20250214_071652.mp4

In the view you'll also see that the sample gets shifted by the border size. I will need to do something about it.

EDIT: Ok I have no idea what happened but when I reopened lmms to work on the issue, it's gone (?) lol what Oh no it's still there, but with a trigger condition

Cool, then, I'll test the things I listed!

As for the clip overlap issue, I don't know, that's too Qt-technical for me to think about.

(Side note, but) I don't even think you can push a clip of your choosing to the front.

If it can be solved so the text is always at the start of the sample, independent of clips that overlap with it, that's great! If not, we can name that as an issue for a future PR to fix in the final report of this PR.

@khoidauminh
Copy link
Contributor Author

Hmmm, I'll try to crash lmms locally. Finding a few hour long samples to concatenate

@khoidauminh
Copy link
Contributor Author

Using a 4 hour long file:

ffprobe version n7.1 Copyright (c) 2007-2024 the FFmpeg developers
  built with gcc 14.2.1 (GCC) 20240910
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-frei0r --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libdav1d --enable-libdrm --enable-libdvdnav --enable-libdvdread --enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgsm --enable-libharfbuzz --enable-libiec61883 --enable-libjack --enable-libjxl --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libplacebo --enable-libpulse --enable-librav1e --enable-librsvg --enable-librubberband --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpl --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-nvdec --enable-nvenc --enable-opencl --enable-opengl --enable-shared --enable-vapoursynth --enable-version3 --enable-vulkan
  libavutil      59. 39.100 / 59. 39.100
  libavcodec     61. 19.100 / 61. 19.100
  libavformat    61.  7.100 / 61.  7.100
  libavdevice    61.  3.100 / 61.  3.100
  libavfilter    10.  4.100 / 10.  4.100
  libswscale      8.  3.100 /  8.  3.100
  libswresample   5.  3.100 /  5.  3.100
  libpostproc    58.  3.100 / 58.  3.100
Input #0, ogg, from 'kdm_discography.ogg':
  Duration: 04:19:52.60, start: 0.000000, bitrate: 127 kb/s
  Chapters:
    Chapter #0:0: start 0.000000, end 221.333000
    Chapter #0:1: start 221.333000, end 437.429000
    Chapter #0:2: start 437.429000, end 1412.862000
    Chapter #0:3: start 1412.862000, end 15592.597000
  Stream #0:0: Audio: vorbis, 48000 Hz, stereo, fltp, 128 kb/s
      Metadata:
        ENCODER         : libaudio: vorbisenc

ảnh

Haven't got a crashed yet

@bratpeki
Copy link
Member

I believe the crash mainly happened the moment I tried playing the track.

@khoidauminh
Copy link
Contributor Author

I played the sample at random places but still couldn't get a crash. I'm using Archlinux so it probably disables memory protection measures or I somehow turned them off.

What if you try to reproduce this in a fresh system?

@sakertooth
Copy link
Contributor

Sometimes when it says "Killed", it might not signify that LMMS has crashed, but something else on the system killed it. I don't know if using an AppImage changes whether or not it prints "Segmentation fault (core dumped)" instead though. @bratpeki, I'm curious if you could try a compiled build and see if anything changes.

@bratpeki
Copy link
Member

bratpeki commented Mar 1, 2025

GDB didn't assist too much:

...
[New Thread 0x7fffdb7ff6c0 (LWP 60170)]
[New Thread 0x7fffdaffe6c0 (LWP 60171)]
[New Thread 0x7fffda7fd6c0 (LWP 60172)]
[Thread 0x7fffda7fd6c0 (LWP 60172) exited]
[Thread 0x7fffdb7ff6c0 (LWP 60170) exited]
[Thread 0x7fffdaffe6c0 (LWP 60171) exited]
[Thread 0x7fffe8dfe6c0 (LWP 60169) exited]
[New Thread 0x7fffda7fd6c0 (LWP 60173)]
[New Thread 0x7fffdaffe6c0 (LWP 60174)]
[New Thread 0x7fffdb7ff6c0 (LWP 60175)]
Lv2 plugin SUMMARY: 0 of 0  loaded in 0 msecs.
[New Thread 0x7fffe8dfe6c0 (LWP 60194)]
[New Thread 0x7fffd95ff6c0 (LWP 60195)]
[New Thread 0x7fffd8dfe6c0 (LWP 60196)]
[New Thread 0x7fffbbfff6c0 (LWP 60197)]
[New Thread 0x7fffbb7fe6c0 (LWP 60198)]
[New Thread 0x7fffbaffd6c0 (LWP 60199)]
[New Thread 0x7fffba7fc6c0 (LWP 60200)]
[Thread 0x7fffba7fc6c0 (LWP 60200) exited]
[Thread 0x7fffbb7fe6c0 (LWP 60198) exited]
[Thread 0x7fffbbfff6c0 (LWP 60197) exited]
[Thread 0x7fffd8dfe6c0 (LWP 60196) exited]
[Thread 0x7fffd95ff6c0 (LWP 60195) exited]
[Thread 0x7fffe8dfe6c0 (LWP 60194) exited]
[Thread 0x7fffdb7ff6c0 (LWP 60175) exited]
[Thread 0x7fffdaffe6c0 (LWP 60174) exited]
[Thread 0x7fffda7fd6c0 (LWP 60173) exited]
[Thread 0x7fffe95ff6c0 (LWP 60095) exited]
[Thread 0x7ffff15fa6c0 (LWP 60076) exited]
[Thread 0x7ffff1dfb6c0 (LWP 60075) exited]
[Thread 0x7ffff5b828c0 (LWP 60035) exited]
[Thread 0x7fffbaffd6c0 (LWP 60199) exited]
[New process 60035]

Program terminated with signal SIGKILL, Killed.
The program no longer exists.
(gdb)

The issue happens when attempting to play with this really large file loaded in. It creates an infobox with messed up drawing and exits:

2025-03-01-134430_1588x868_scrot


I still believe this is a fix for a future PR, though, and if @khoidauminh would be interested in investigating this, I would be more than greatful!

@bratpeki
Copy link
Member

bratpeki commented Mar 1, 2025

When loading samples which take a bit of time, that infobox says "Sample loading/This may take a bit of time..."

Sometimes, this info is displayed properly, sometimes not. Can anyone replicate this?

@bratpeki
Copy link
Member

bratpeki commented Mar 1, 2025

2025-03-01.14-04-12.mp4

A little video demonstration.

@bratpeki
Copy link
Member

bratpeki commented Mar 1, 2025

In any case, as this is also on the nightly, it's up to a future fix!

Right, so I should expect the things listed below, and if they're accomplished, this has been tested?

  • Any sample, no matter the length or zoom level, now displays properly
  • The part of the waveform between -1.0 and 0.0 isn't filled anymore

Both fixed!

@bratpeki bratpeki self-requested a review March 1, 2025 13:10
@bratpeki
Copy link
Member

bratpeki commented Mar 1, 2025

@sakertooth, please take a look and merge if all is well. I'll open the corresponding issues on the latest nightly once this is merged.

Copy link
Contributor

@sakertooth sakertooth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code changes look fine. Will test soon.

Copy link
Contributor

@sakertooth sakertooth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Garbage rendering was fixed, thumbnails that exactly matched the original sample data are no longer generated, and some more. Fine with me. Thanks @khoidauminh for the PR and @bratpeki for the testing. We should be good if there isn't any other issue we need to address.

@bratpeki
Copy link
Member

bratpeki commented Mar 2, 2025

Garbage rendering is on master for me, so there's nothing left to address! Please merge 🚀

@sakertooth
Copy link
Contributor

sakertooth commented Mar 2, 2025

Should resolve the sample clips part of #3378

Can we consider this issue fixed @khoidauminh? The issue specifically mentioned sample clips so I think it is safe to make that assumption.

@khoidauminh
Copy link
Contributor Author

@sakertooth We still have glitched rendering with other clips so I don't think we can consider the entire issue fixed yet. I was thinking about making an attempt to fix the remaining ones once this PR is merged. I suppose @zonkmachine can modify #3378 to track which types of clips have been resolved.

@zonkmachine
Copy link
Member

I suppose @zonkmachine can modify #3378 to track which types of clips have been resolved.

I can do that.

@bratpeki
Copy link
Member

bratpeki commented Mar 3, 2025

We still have glitched rendering with other clips

Could you elaborate on that a bit? From my understanding, this generates the clip thumbnail based on how much clip is currently visible, so there being garbage rendering is a bit confusing to me. 😅

@khoidauminh
Copy link
Contributor Author

khoidauminh commented Mar 3, 2025

@bratpeki This PR fixes the garbage rendering for sample clips, but midi, pattern and automation clips still suffer from it

@bratpeki
Copy link
Member

bratpeki commented Mar 3, 2025

I see. It's out of scope for this PR, though, no?

@@ -105,6 +103,7 @@ class LMMS_EXPORT SampleThumbnail

Thumbnail zoomOut(float factor) const;

Peak* data() { return m_peaks.data(); }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I added this, but can you make it return const Peak* for me instead?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait nevermind I see there was an issue somewhere. Let me review a bit more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the dereferencing logic was solid, it's odd how it appeared. I think it must be some problem with the bounds. I got this crash too I believe so I'll confirm if the fix works or not.

@sakertooth
Copy link
Contributor

I see. It's out of scope for this PR, though, no?

Yeah, most likely. It can be dealt with later in a PR with a clear focus on fixing it for the other clip types.

@sakertooth
Copy link
Contributor

How hard would it be to add tests for sample thumbnails? To minimize the chance of any bugs and regressions.

@khoidauminh
Copy link
Contributor Author

khoidauminh commented Mar 5, 2025

How hard would it be to add tests for sample thumbnails? To minimize the chance of any bugs and regressions.

Do you mean testing in general or automated tests?

I usually test in 4 places: Song editor, AFP, SliceT and Automation editor. Actions usually include, zooming all the way in, reversing, sliding, scrolling, overlaying another clip/window on top and trigger repaints, etc

TBH, I think the best way to test this is to roll it out to master, where it's exposed to all nightly users. The master branch is for development in general, but it's definitely used catching bugs/regressions as well.

@sakertooth
Copy link
Contributor

I mean automated testing. Our testing suite is really lacking right now, and with more features being added without tests to cover our mistakes, it's the equivalent of just winging it and hoping there's no problems.

@sakertooth
Copy link
Contributor

I don't even think we can work towards a stable release without tests.

@sakertooth
Copy link
Contributor

sakertooth commented Mar 5, 2025

The master branch is for development in general, but it's definitely used catching bugs/regressions as well.

I think that the dereference bug could've been caught in an automated testing suite, as long as it's comprehensive enough. That way, if new changes are applied, it will always check against it, whereas with manual testing the tester may not be interested in doing a comprehensive test if the change seems localized in one place, and they don't consider potential side effects.

How can we reach a stable release if we are dealing with regressions all the time? The problem is we could check off that a bug was fixed, but when another PR comes around and brings the bug out of dormancy again, it's basically whack-a-mole software development.

@bratpeki
Copy link
Member

bratpeki commented Mar 5, 2025

@sakertooth I'd recommend opening a project for automated tests and solving it one issue at a time. Hell, add every single one of them to the 1.3 milestone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants