Stream API Vulkan Backend #9164

phoenixxxx · 2025-08-29T17:37:01Z

This is the implemenation of the Stream API on the Vulkan Backend. We leverage most of the work that was done by the external image manager, but we created a different manager to contain the code.

Tested on Moohan (manual merge, so some things could have gotten lost in the process). Also working on validating this on Android.

filament/backend/src/vulkan/VulkanStreamedImageManager.cpp

filament/backend/src/vulkan/VulkanDriver.cpp

filament/backend/src/vulkan/VulkanTexture.h

filament/backend/src/vulkan/VulkanStreamedImageManager.cpp

rafadevai · 2025-09-03T16:38:43Z

filament/backend/src/vulkan/VulkanDriver.cpp

@@ -455,13 +456,18 @@ void VulkanDriver::updateDescriptorSetTexture(
    if (UTILS_UNLIKELY(mExternalImageManager.isExternallySampledTexture(texture))) {
        mExternalImageManager.bindExternallySampledTexture(set, binding, texture, params);
        mAppState.hasBoundExternalImages = true;
-    } else {
+    } else if (bool(texture->getStream())) {


can a texture be external sampled and streamed at the same time? or are they mutually exclusive?

Logically yes. But that would mean that a texture that is external was attached to a stream. And that seems like something went wrong, because from the frontend's perspective you can create a texture one of 3 ways:

From a format

From an external image

By setting a stream
Although you could logically do 2) and then set a stream via 3, I question the meaning of that.

filament/backend/src/vulkan/VulkanStreamedImageManager.h

poweifeng · 2025-09-02T17:56:40Z

filament/backend/include/backend/platforms/VulkanPlatformAndroid.h

    ExternalImageHandle UTILS_PUBLIC createExternalImage(AHardwareBuffer const* buffer,
            bool sRGB) noexcept;
+    virtual ExternalImageHandle createExternalImageFromRaw(void* image,
+            bool sRGB) noexcept override;


I think we should just combine these methods into one called createExternalImage(void* image, bool sRGB)

Yeah this issue is that Mathias didn't want to do that.
The whole concept of ExternalImageHandleRef exists to avoid using raw pointer. Something that the Stream API totally breaks, it relies (on the GL and VK sides) on void*.

Second issue is we did a ton of changes on the Impress side to create, and send ExternalImageHandleRef (works for all external image types) so undoing this would need many Impress side changes.

I don't even understand what the void* is here? This needs to be documented.

So that's because in the stream case, we call createExternalImageFromRaw insinde the driver when we process the pending stream. And inside VulkanDriver.cpp we don't know what the platform is, so we can't assume createExternalImage(AHardwareBuffer const*)

Ok I see why you need to do this. This is because AcquiredImage has a void*. I think this is okay for now (for the sake of getting this done). But really, we should change the Acquired Stream API to only speak in terms of ExternalImageRef.

HOWEVER, I think this means that createExternalImageFromRaw() must not be public. It must be protected. And the concrete implementations must be private. This is needed only for internal machinery.

Yes I am ok with that. Frankly, I think I agree, the stream taking void* isn't the right call.

How do I not make this public?

filament/backend/src/vulkan/VulkanDriver.cpp

poweifeng · 2025-09-02T18:02:47Z

filament/backend/src/vulkan/VulkanTexture.h

@@ -36,6 +36,34 @@ namespace filament::backend {

 struct VulkanTexture;

+struct VulkanStream : public HwStream, fvkmemory::Resource {


This should be fvkmemory::ThreadSafeResource

I think I've found a way around this.

What way did you find?

So first the lifetime of the resource is only managed on the frontend (ie: the backend doesnt hold resources to the stream) so we don;t need to atomic incredment of the ThreadSafeResource. Stream is pretty much like a texture or any other fvkmemory::Resource derived class.

This follows the desing pattern of GL and like GL in updateStreams we just make sure to capture and access parts of the stream that the front end doesnt touch.

Backend does in fact hold all the resources. This is due to the ref-counting mechanism of the vk backend. This is not true for the GL backend. This does have to be a ThreadSafeResource because the resource is accessed in both the backend thread and the frontend thread. And ThreadSafeResource cannot hold references to other resources due to this

poweifeng · 2025-09-02T18:03:24Z

filament/backend/src/vulkan/VulkanTexture.h

+private:
+    AcquiredImage mAcquired;
+    AcquiredImage mPrevious;
+    std::map<void*, fvkmemory::resource_ptr<VulkanTexture>> mTextures;


ThreadSafe resources can't have any resource_ptr fields unfortunately. Maybe you can keep this map in VulkanStreamedImageManager

I think i've found a way around this.

filament/backend/src/vulkan/VulkanTexture.h

filament/backend/src/vulkan/VulkanDriver.cpp

pixelflinger · 2025-09-04T20:12:25Z

filament/backend/include/backend/platforms/VulkanPlatform.h

    virtual ImageData createVkImageFromExternal(ExternalImageHandleRef image) const {
        return {};
    }
+    virtual ExternalImageHandle createExternalImageFromRaw(void* image, bool sRGB) noexcept {
+        return {};
+    }


all virtuals in this file should not be implemented in the header file

all need documentation, as this is the public API.

pixelflinger · 2025-09-04T20:14:54Z

filament/backend/include/backend/platforms/VulkanPlatformAndroid.h

    ExternalImageHandle UTILS_PUBLIC createExternalImage(AHardwareBuffer const* buffer,
            bool sRGB) noexcept;
+    virtual ExternalImageHandle createExternalImageFromRaw(void* image,
+            bool sRGB) noexcept override;


I don't even understand what the void* is here? This needs to be documented.

filament/backend/src/vulkan/VulkanDriver.cpp

rafadevai · 2025-09-05T17:42:03Z

filament/backend/src/vulkan/VulkanDriver.cpp

+                    copyMat3f(bd.buffer, offset, transform);
+                }
+            }
+            mStreamUniformDescriptors.erase(streamDescriptors);


why do we have to erase this?

So the logic is that the user has to submit an update to the stream descriptor every time the xform changes. Once we process it, we can remove it from the list.

pixelflinger · 2025-09-08T07:01:47Z

filament/backend/include/backend/platforms/VulkanPlatformAndroid.h

    ExternalImageHandle UTILS_PUBLIC createExternalImage(AHardwareBuffer const* buffer,
            bool sRGB) noexcept;
+    virtual ExternalImageHandle createExternalImageFromRaw(void* image,
+            bool sRGB) noexcept override;


Ok I see why you need to do this. This is because AcquiredImage has a void*. I think this is okay for now (for the sake of getting this done). But really, we should change the Acquired Stream API to only speak in terms of ExternalImageRef.

HOWEVER, I think this means that createExternalImageFromRaw() must not be public. It must be protected. And the concrete implementations must be private. This is needed only for internal machinery.

filament/backend/src/vulkan/VulkanDriver.cpp

poweifeng · 2025-09-08T17:42:59Z

filament/backend/src/vulkan/VulkanDriver.cpp

+            // executes on the user thread) Note: stream is captured by copy which is fine, this is
+            // a copy of a resource_ptr<VulkanStream>. We only need it find the associated stream
+            // inside the mStreamedImageManager texture bindings
+            driver->queueCommand([this, stream, s = stream.get(),


why do the s=stream.get?

Because it's in the loop, and the stream variable will actually change, so I think we need a unique copy of the stream pointer (access underlying data) per queueCommand

So I think stream is a copy of the ref-counted pointer to the actual content already. You shouldn't need to do s = stream.get (unless you pass it with &stream, then it's a reference. But even this is ok I think).

in which case will change? Now that i look at this again, do you actually need stream and s at the same time?

Yeah so this follows the GL design pattern where it gets the underlying raw stream object and sends it to the backend (the backend only manipulates the raw data thereby ensure we don't need to share the actual VulkanStream object betwen FE and BE... I can explain this on the chat, it's a bit tricky (not great) but that's what the GL path does.

in which case will change? Now that i look at this again, do you actually need stream and s at the same time?
This is because we use the VulkanStream (not VulkanStream*)

I still don't understand why you can't just pass stream by copy. stream is a pointer. It can be dereferenced to get the content (s).

filament/backend/src/vulkan/VulkanDriver.cpp

poweifeng · 2025-09-08T17:54:24Z

filament/backend/src/vulkan/VulkanDriver.h

+    VulkanStreamedImageManager mStreamedImageManager;
+
+    // Stream transforms
+    std::unordered_map<VkBuffer, BufferObjectStreamDescriptor> mStreamUniformDescriptors;


can we keep this in VulkanStreamedImageManager ?

I would rather not. It feels more nature for the stream to hold this data. It also means that once we have the stream we can access all the stream images. Since this object is not a Thread safe resource, it seems sensible?
But if you feel strongly about it, I can move it to the VulkanStreamedImageManager, that's one more level of indirection where we need a to map<VkBuffer, BufferObjectStreamDescriptor> inside the VulkanStreamedImageManager

filament/backend/src/vulkan/VulkanDriver.cpp

filament/backend/src/vulkan/VulkanStreamedImageManager.cpp

filament/backend/src/vulkan/VulkanTexture.h

poweifeng · 2025-09-09T15:42:43Z

filament/backend/src/vulkan/VulkanDriver.cpp

+            // executes on the user thread) Note: stream is captured by copy which is fine, this is
+            // a copy of a resource_ptr<VulkanStream>. We only need it find the associated stream
+            // inside the mStreamedImageManager texture bindings
+            driver->queueCommand([this, stream, s = stream.get(),


So I think stream is a copy of the ref-counted pointer to the actual content already. You shouldn't need to do s = stream.get (unless you pass it with &stream, then it's a reference. But even this is ok I think).

filament/backend/src/vulkan/VulkanDriver.cpp

poweifeng · 2025-09-09T15:45:43Z

filament/backend/src/vulkan/VulkanStreamedImageManager.cpp

+                // Eventually the updateSampler and updateSamplerForExternalSamplerSet
+                // will call to vkUpdateDescriptorSets with a VkWriteDescriptorSet
+                // type VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER the
+                // VkDescriptorImageInfo will contain the view of the new image frame.
+                mDescriptorSetCache->updateSamplerForExternalSamplerSet(data.set, data.binding,
+                        image);


I don't think you need this part even. This will be done in ExternalImageManager::prepareBindSets.

Ah yes, I think this is right. bindExternallySampledTexture pushes into mSetBindings and we allways update all the bindings.

poweifeng · 2025-09-09T15:48:03Z

filament/backend/src/vulkan/VulkanStreamedImageManager.cpp

+            // For some reason, some of the frames coming to us, are on streams where the
+            // descriptor set isn't external...
+            if (data.set->getExternalSamplerVkSet()) {
+                if (newImage) {


I think we can always just call bindExternallySampledTexture even if it's not a newImage ?

But we should probably add some logic in bindExternallSampledTexture to remove an entry if it shares the same data.set and data.binding. (Basically it's overwriting the existing binding for a descriptor set).

I don't think so, we get these calls every frame, which means we will keep pushing int the binding vector and it will grow unbounded.

poweifeng · 2025-09-09T15:49:28Z

filament/backend/src/vulkan/VulkanTexture.h

@@ -36,6 +36,34 @@ namespace filament::backend {

 struct VulkanTexture;

+struct VulkanStream : public HwStream, fvkmemory::Resource {


What way did you find?

phoenixxxx and others added 30 commits June 4, 2024 12:35

Adding the begin frame message for later xtrace post processing.

6218164

Merge branch 'main' into main

e265850

Merge branch 'main' into main

8671617

Merge branch 'main' into main

d49554d

Merge branch 'google:main' into main

4fccd95

Merge branch 'google:main' into main

96cba96

Merge branch 'google:main' into main

abe8a49

Merge branch 'google:main' into main

c81cedd

Merge branch 'google:main' into main

387942c

Merge branch 'google:main' into main

0fdf1ca

Merge branch 'google:main' into main

8bc4d3a

Merge branch 'google:main' into main

9e92193

Merge branch 'google:main' into main

01a2480

Merge branch 'google:main' into main

48bb10a

Merge branch 'google:main' into main

ff145f6

Merge branch 'google:main' into main

c5c2d42

Merge branch 'google:main' into main

3b0e44c

Merge branch 'google:main' into main

80803f9

Merge branch 'google:main' into main

ad3a5b7

Merge branch 'google:main' into main

be6b6ae

Merge branch 'google:main' into main

021c4f6

Merge branch 'google:main' into main

0a661c1

Merge branch 'google:main' into main

ea4c20e

Merge branch 'google:main' into main

ba99844

Merge branch 'google:main' into main

cfe6a42

Merge branch 'google:main' into main

7f7dadd

Merge branch 'google:main' into main

b570d48

Merge branch 'google:main' into main

b09cbaf

Merge branch 'google:main' into main

13e1d2a

Merge branch 'google:main' into main

ff2827b

phoenixxxx and others added 4 commits August 29, 2025 10:24

Refactoring of the code.

0c07c3f

Merge branch 'google:main' into main

f503205

Merge branch 'main' into streamed_texture

6e8959e

Typo.

39be20b

phoenixxxx added the internal Issue/PR does not affect clients label Aug 29, 2025

phoenixxxx changed the title ~~Streamed API Vulkan Backend~~ Stream API Vulkan Backend Aug 29, 2025

rafadevai reviewed Sep 3, 2025

View reviewed changes

filament/backend/src/vulkan/VulkanStreamedImageManager.cpp Outdated Show resolved Hide resolved

rafadevai reviewed Sep 3, 2025

View reviewed changes

filament/backend/src/vulkan/VulkanDriver.cpp Show resolved Hide resolved

rafadevai reviewed Sep 3, 2025

View reviewed changes

filament/backend/src/vulkan/VulkanDriver.cpp Outdated Show resolved Hide resolved

rafadevai reviewed Sep 3, 2025

View reviewed changes

phoenixxxx requested review from poweifeng and rafadevai September 4, 2025 02:48

phoenixxxx and others added 3 commits September 3, 2025 20:14

GitHub feedback.

db83dc0

Merge branch 'google:main' into main

54c636d

Merge branch 'main' into streamed_texture

562eeed

poweifeng reviewed Sep 4, 2025

View reviewed changes

pixelflinger requested changes Sep 4, 2025

View reviewed changes

phoenixxxx added 3 commits September 4, 2025 19:57

Github feedback

74ffe4e

feedback.

32ca1d3

Typo

3457700

rafadevai reviewed Sep 5, 2025

View reviewed changes

filament/backend/src/vulkan/VulkanDriver.cpp Outdated Show resolved Hide resolved

rafadevai reviewed Sep 5, 2025

View reviewed changes

pixelflinger requested changes Sep 8, 2025

View reviewed changes

poweifeng reviewed Sep 8, 2025

View reviewed changes

phoenixxxx added 2 commits September 9, 2025 08:16

Feedback.

85c899a

Feedback.

0ba9c2a

poweifeng reviewed Sep 11, 2025

View reviewed changes

phoenixxxx added 2 commits September 11, 2025 12:34

Feedback

78d5e34

Feedback

1216024

		@@ -36,6 +36,34 @@ namespace filament::backend {

		struct VulkanTexture;

		struct VulkanStream : public HwStream, fvkmemory::Resource {

Stream API Vulkan Backend #9164

Are you sure you want to change the base?

Stream API Vulkan Backend #9164

Conversation

phoenixxxx commented Aug 29, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!