Recording and Playing back a lot of floats

vvvv beta general
1

Hi All

On the project I am currently working on I need to record the contour data coming out of the contour node.

I have tried recording to a text file and playing back from it again.
It is not surprising stuttering heavily and considering that it is just relatively few numbers compared to recording and playing back eg. video I am thinking there must be a more efficient method. of course I can optimize by cropping values to the needed precision so I don’t get a lot of decimals.

So My question to the forum is if you have a more efficient way of recording a lot of floats? I am thinking of some sort of binary format. I have been thinking about using a database but is thinking that it can’t be optimal to wrap the whole thing into sql.
So any Ideas?

sunep

2

ai sune, eno would recommend you to write the data to a dynamictexture.

3

Hello,

just out of interest, how long is your recording?

Having a custom binary format is not necessarily faster using an sql database, specially a file based one (like sqlite) which are pretty good at indexing.

If you need more info let me know.

4

Hi joreg and vux

@vux I am still a little reluctant to use the database approach, my main concern is the overhead both in cpu and size from converting the floats to strings. But should give it a try. regarding the length of the recording I tried making one around 10 minutes and at 30fps the resulting textfile got pretty large, many megabytes.

@joreg I have been playing a bit around with the dynamic texture approach and it seems like I could get it to work, I do however have some doubts about if it will be a good approach. If using this method I would need to record the changing texture as video and a codec would then be used to compress it, I am not sure how that would affect the precision of the data when replayed. further more the additional trouble of having integers larger than 255 is a bit complex. also the fact that the number of points varies which makes it necessary to use a relatively large texture.

I can see how both approaches can solve the problem, but I still feel that a more elegant and optimal solution could be possible. in any case I would like to wish for a method of saving floats and perhaps even other datatypes in what I imagine would be a binary format so I would not have to convert to strings and thus increase the amount of data saved.

any other ideas how to achieve this?

5

Hey sunep

my main concern is the overhead both in cpu and size from >converting the floats to strings

If you run a query per frame (which will be executed in no time), you’ll have let’s say maximum 500 conversions, which is negligible to be fair. I would still not use a network database like mysql, that’s way overkill, small file based system will definitely do the job.

The other option is to write in a binary format and do your own indexing (i did that for a FFT recorder), but that means some plugin coding in that case.
Your case will be a tiny bit more complicated than mine as on each frame the number of contour can change, so the data size per frame can be different (whereas fft stays the same all the time). It’s not too hard still, but need some thoughts.

By the way do you record all the contour data, or just the center of the blobs?

6

@vux

I record the contour data themselves… usually around 1000 points/frame. and then the BinSizes.

At first I was looking a bit into mysql, but as you say it is overkill. I will try sqlite out.

thanks for the advice

7

you have sqlite support in the addon pack.

Have fun.

8

Prioritized this down a bit to finish other stuff…

But now I am looking into sqlite and there are some details that I wonder a bit about:

How do I create tables? am I correct that I need to create a database with tables outside of vvvv and then access it from vvvv? can I create tables from vvvv?

Perhaps it is just me being newbie in accessing my own databases that makes me wonder, so far saving things as xml has been ok but I need a more effective way of doing this for this particular case.

I have tried finding something on databases and vvvv, but it is a little difficult. And the help patches does not deal with creating the database itself.

9

Hello,

for sqllite, just put the filename you want, it will create the file for you if it doesn’t exists.

To create tables, QueryBatch (Sqlite Network) allows you to run any statement(s), so you can use this one.

10

Ah ok, that makes sense.

Would it be ok if I modify the help patches, just to have references to the other sqlite files.

then I would also need to figure out how to upload them, but I would not mind doing that.

sune

11

Now I am testing recording… it really halts the computer, I achieve between 7 and 10 FPS when using sqlite for recording. it seems like playing back the data is no problem though so having the data in sqlite.

I have tried reducing the length of the strings written to the database by converting the floats into hex using radix and then into string, then the strings are aprox. half the length compared to just converting the regular floats to string.

at the 30 fps the camera is running I would like to record a string around 3500 chars long. so in all ideally around 105000bytes a second.

any ideas how to record this faster than this?

Sune