Public API for AI-powered patching

Since there has been a few chats about gamma + AI- coding, and there is this question of how gamma can keep up with other IDEs, most of which are fully AI-powered by now, I believe it’s worthwile to explore MCP servers (Introduction - Model Context Protocol).

Basically, MCP servers enable LLM chatbots to run programs, eg run shell commands, surf on the web, etc. There’s already a lot of them for all kinds of programs or services, also community ones eg for Ableton or Touchdesigner.

So a vvvv-mcp-server could enable an ai-powered tool like Claude Desktop to create/run/edit/explain/debug vvvv gamma patches. Newer agent-like bots, which can evaluate outputs (eg screenshots, console, etc.) and then re-iterate could setup complete apps this way. In theory, this could also be built into gamma in the long run.

The interesting thing about this is that it’s not necessary to do extra training of a LLM on vl xmls or anything like that, instead, the mcp server “only” needs to understand how a vl xml is structured, have access to a searchable list of available nodes with inputs, outputs, info, etc. as well as to a knowledgebase (a structured searchable database of docs, help file xmls, etc…).
It’s rather crucial to have this info well-structured as to minimize the context the LLM has to take in per step it has to execute.


Now the most straightforward/cleanest way to get there would be if gamma would expose a public api, which enables to create/save/etc. patches, browse nodes and help files, search for connectable nodes, read console output etc…this is the actual feature request of this post. I’m sure a nice public api for vvvv would come in handy in other situations as well.


However, even without an api, its entirely possible to hack this.

  • creating patches or nodes, saving, editing, etc. is entirely done on xml basis, no gamma editor needed
  • run, restart commands etc. are done by simulating a keyboard
  • websocket or similar can be used to communicate things like console output
  • the necessary searchable knowledgebase containing graybook info and help files can be compiled by a llm
  • a list of available nodes can be compiled from the vvvv/pack and nuget folder. (I have already prototyped this and have a c# app which will read all usable nodes from a nuget. I can’t correctly read all input/output types yet, unless they are typed in vvvv. If their type is inferred from the patch I (->Claude) would have to code some kind of crawler which tries to figure out the type. But since vvvv does all of this already I’m wondering now if it’s possible to retrieve all available nodes from within vvvv as some kind of List?)

Thoughts?

9 Likes

Nice idea,I am very expect it !

From what I can tell the quickest starting point would be to run vvvv with the visual studio debugger that will support MCP. There a preview available right now, but I haven’t tried it, nor Claude Code, or anything else with MCP for that matter, but maybe it give the devs something to explore. Along with an extension could be made as a server exposing the node information for the patch (checkout the help patch with the search ‘API’).

That might be enough to get something like code patch suggestions.
There is the IL’ed code Window that updates automatically, but I don’t know how that works. If an AI helper could get a look at that it real time, plus the outputs from a debugger, maybe that would give it a better idea what’s going on.

I’ve never dare look inside before but among the Nugets (not VL Nugets) I can see that VL.AppServices which looks like in could handle information and some of the behaviour of the IDE. If there were some documentation or xml comments I wouldn’t mid diving in.

I don’t have much time to look at it these days but I would love the devs to chime in on this. @gregsn ?