Hello.
I need to get data from TCP, which is huge, so I need to append a lot of Raw values to big buffer. How can I achieve this?
It should work like Writer (Raw) node, but it should just keep data in memory.
I tried using Queue and then + (Spectral), but vvvv crashes on it (and not because of memory overflow issues).
I am thinking about writing new node using Template (Raw). Is there any better solution?
what’s the problem with appending to a temporary file?
It works, but why do I need disk I/O when all my data fits in memory?
So problems:
can you patch us a demo that shows this?
then you can also try Writer (Raw SharedMemory) that will be available with upcoming beta32. only it needs a max size set from the beginning.
Sure. Attached small example.
So what do you recommend me right now?
To write plugin using Template (Raw)?
patch with reproducible bug (793.8 kB)
i just tried your patch and while it crashes with b31.2 it works fine with b32.
works alright in beta32, thank you.
The fixed size is quite okay to work with, but I think the node should be doing some internal auto-padding. Right now it overwrites the shared memory only with as much bytes as are inputted and leaves the rest as it is.
One patch-around is to use Pad (String) but I don’t think that is the smart choice for most cases involving Raw. The other is to + (Raw) with a long-enough bytestream, which is quite cryptic to read.
@velcrome what other benefit than having a clean memory space (which noone ever sees) would such an auto-padding have?
this is only a matter when bytestreams with changing length are sent through a fixed sized transmitter.
on the receiving end it is impossible to tell, which data is actually fresh or useful and what part is just left over from frames long gone.
padding kind of solves this by making sure, there are no left overs at all.
in any case i’d argue you’d better reserve the first 1-4 bytes to encode the length of the current data and better rely on that than to look for some padding at the end.
I agree, a length index would be even better. Is this something that could be put inside the plugin?
that could be a job for a module around the node then…