Since there has been a few chats about gamma + AI- coding, and there is this question of how gamma can keep up with other IDEs, most of which are fully AI-powered by now, I believe it’s worthwile to explore MCP servers (Introduction - Model Context Protocol).
Basically, MCP servers enable LLM chatbots to run programs, eg run shell commands, surf on the web, etc. There’s already a lot of them for all kinds of programs or services, also community ones eg for Ableton or Touchdesigner.
So a vvvv-mcp-server could enable an ai-powered tool like Claude Desktop to create/run/edit/explain/debug vvvv gamma patches. Newer agent-like bots, which can evaluate outputs (eg screenshots, console, etc.) and then re-iterate could setup complete apps this way. In theory, this could also be built into gamma in the long run.
The interesting thing about this is that it’s not necessary to do extra training of a LLM on vl xmls or anything like that, instead, the mcp server “only” needs to understand how a vl xml is structured, have access to a searchable list of available nodes with inputs, outputs, info, etc. as well as to a knowledgebase (a structured searchable database of docs, help file xmls, etc…).
It’s rather crucial to have this info well-structured as to minimize the context the LLM has to take in per step it has to execute.
Now the most straightforward/cleanest way to get there would be if gamma would expose a public api, which enables to create/save/etc. patches, browse nodes and help files, search for connectable nodes, read console output etc…this is the actual feature request of this post. I’m sure a nice public api for vvvv would come in handy in other situations as well.
However, even without an api, its entirely possible to hack this.
- creating patches or nodes, saving, editing, etc. is entirely done on xml basis, no gamma editor needed
- run, restart commands etc. are done by simulating a keyboard
- websocket or similar can be used to communicate things like console output
- the necessary searchable knowledgebase containing graybook info and help files can be compiled by a llm
- a list of available nodes can be compiled from the vvvv/pack and nuget folder. (I have already prototyped this and have a c# app which will read all usable nodes from a nuget. I can’t correctly read all input/output types yet, unless they are typed in vvvv. If their type is inferred from the patch I (->Claude) would have to code some kind of crawler which tries to figure out the type. But since vvvv does all of this already I’m wondering now if it’s possible to retrieve all available nodes from within vvvv as some kind of List?)
Thoughts?