Memory Issues when loading Textures on the Fly

[lunanane]

Hello,

I wanted to revisit this topic, that was outlined in details by @readme in this thread a while ago:

Selectively preloading textures in the background

here is one more idea, try loading the textures one by one with some time in between, loading one at a time should not hit the main loop that much, as seen in the ImagePlayer.

Simmilar to the problems in this thread, i am having an application that continously need to load from a large amount of textures on a drive.
I am loading 64 times 1024x1024 res texture in a for loop and rendering them to basic meshrenderer drawcalls. And the textures are changing often during runtime.

The problem is, that FileTexture is the node that is producing the least lag, however it is keeping the texture in memory. This is useful in this scenario because then the texture is directly available when its loaded a second time.
The problem is that the RAM is filling up quiete fast. Only when it is maxing out at 64 GB, it is starting to have some kind of garbage collection process in the background, causing it to go down with RAM to about under 55 GB, but then it would hover with this high RAM through out the runtime of the application, making it instable as soon as a browser is opened or any other task performed on the system! It might also stop loading textures at all after additional Ram is needed from the system.

So the question here is, can we limit RAM usage of the underlying Stride engine?

The second problem I have is that as an alternative TextureReader (Async) was proposed in another thread.
However this node is producing a lag when used in a for loop region and is significantly slower, so i can not use it (at least not in the same instance, and i am refusing so far to try piping continusly 64 textures with spout and ending up with a complicated loading logic)

As a workaround to the for loop lag, I tried to build a sequential loader that uses TextureReader (Async) and a Queue node with a respective loading logic, but it is unfortunatly too slow even on the fastest NVME SSD.

The ImageReader Skia Node is much fast and could be used to fill the sequential loading queue. However, when using SkiaTexture Node. The Queue node has a bug that causes all the slices to be set to the current input texture, instead of only the first slice. This does not happen with FileTexture or any other Texture Input. Reporting this bug is a bit of an extra story, but my point is, that I feel like I am out of choises for effective on the fly loading of a lot of textures??

Any recommendations to this are very appreciated :)

edit: and it is not possible to convert the textures (.png and .jpg) to compressed format because of filesize is huge and it would take probably more than a week

[tonfilm]

Did you try the compressed flag on the file texture? It will create dxt format textures, which will significantly reduce memory usage.

But depending on the amount, you will run into the same issues…

Also note that file texture will use a cache on the dis,k which will grow. It’s in the data folder somewhere. It will help to load the textures instantly the second time they are seen by the application. But here you also have to take care of size/growth over time.

Did you try TextureReader non async in an async region? That will not use the disc cache, and could be set to another thread, and the texture can also be disposed later, and should clear the ram and vram then…

These are some ideas you can try.

[lunanane]

@tonfilm Thanks for the input!

I think its loading already compressed, so yes, same issue.

The cache is this DB file mentioned here i guess?

Lib/data/db

Hello! I am working on a project using stride to reposition, rescale, ect. Filteextures. On that machine the folder vvvv_gamma_2021.4.10 / lib / data / db now has a filesize over 200 GB. Machine will not start any more. Can I delete the whole db folder completly? Please help me with this. Thanks in advance.

Is it save to delete during runtime? I might try and report, but it seems that this is unrelated to RAM, no?

I tried TextureReader non async in async region and it also introduced a lag somehow.. I asume that receiving those textures async is not possible without a lag.. But just a guess. I havent looked in how to asign it to another thread, is this easy to do?

[bjoern]

What exactly does “lag” mean in this context?

[lunanane]

@bjoern the lag means a noticable framedrop when loading.
camera animation is happening in this moment and you see it..

[joreg]

are you aware of this help patch? “HowTo Manage loading and unloading of textures”.

could be improved, but if you for example add LFO and Random like this:

would this come close to your usecase? and does this produce framedrops for you?