Public API for AI-powered patching

vvvv gamma feature
1

Since there has been a few chats about gamma + AI- coding, and there is this question of how gamma can keep up with other IDEs, most of which are fully AI-powered by now, I believe it’s worthwile to explore MCP servers (Introduction - Model Context Protocol).

Basically, MCP servers enable LLM chatbots to run programs, eg run shell commands, surf on the web, etc. There’s already a lot of them for all kinds of programs or services, also community ones eg for Ableton or Touchdesigner.

So a vvvv-mcp-server could enable an ai-powered tool like Claude Desktop to create/run/edit/explain/debug vvvv gamma patches. Newer agent-like bots, which can evaluate outputs (eg screenshots, console, etc.) and then re-iterate could setup complete apps this way. In theory, this could also be built into gamma in the long run.

The interesting thing about this is that it’s not necessary to do extra training of a LLM on vl xmls or anything like that, instead, the mcp server “only” needs to understand how a vl xml is structured, have access to a searchable list of available nodes with inputs, outputs, info, etc. as well as to a knowledgebase (a structured searchable database of docs, help file xmls, etc…).
It’s rather crucial to have this info well-structured as to minimize the context the LLM has to take in per step it has to execute.

Now the most straightforward/cleanest way to get there would be if gamma would expose a public api, which enables to create/save/etc. patches, browse nodes and help files, search for connectable nodes, read console output etc…this is the actual feature request of this post. I’m sure a nice public api for vvvv would come in handy in other situations as well.

However, even without an api, its entirely possible to hack this.

  • creating patches or nodes, saving, editing, etc. is entirely done on xml basis, no gamma editor needed
  • run, restart commands etc. are done by simulating a keyboard
  • websocket or similar can be used to communicate things like console output
  • the necessary searchable knowledgebase containing graybook info and help files can be compiled by a llm
  • a list of available nodes can be compiled from the vvvv/pack and nuget folder. (I have already prototyped this and have a c# app which will read all usable nodes from a nuget. I can’t correctly read all input/output types yet, unless they are typed in vvvv. If their type is inferred from the patch I (->Claude) would have to code some kind of crawler which tries to figure out the type. But since vvvv does all of this already I’m wondering now if it’s possible to retrieve all available nodes from within vvvv as some kind of List?)

Thoughts?

11 Likes
2

Nice idea,I am very expect it !

3

From what I can tell the quickest starting point would be to run vvvv with the visual studio debugger that will support MCP. There a preview available right now, but I haven’t tried it, nor Claude Code, or anything else with MCP for that matter, but maybe it give the devs something to explore. Along with an extension could be made as a server exposing the node information for the patch (checkout the help patch with the search ‘API’).

That might be enough to get something like code patch suggestions.
There is the IL’ed code Window that updates automatically, but I don’t know how that works. If an AI helper could get a look at that it real time, plus the outputs from a debugger, maybe that would give it a better idea what’s going on.

I’ve never dare look inside before but among the Nugets (not VL Nugets) I can see that VL.AppServices which looks like in could handle information and some of the behaviour of the IDE. If there were some documentation or xml comments I wouldn’t mid diving in.

I don’t have much time to look at it these days but I would love the devs to chime in on this. @gregsn ?

2 Likes
4

I am not the most knowledgeable in the topic but would love to learn and get involved where help is needed.

5

From the element chat:

@texone :
Regarding MCP and AI integration because it was brought up yesterday in the meetup at the end again
🚨 MCP + AI Integration is the key topic for vvvv right now

During yesterday’s meetup someone asked about MCP integration (Model Context Protocol) right at the end. I think this deserves much more attention:

This isn’t just another feature – it’s the strategic question for vvvv’s future.

Why?

👉 vvvv set out to make programming radically easier – visual, immediate, exploratory.
👉 AI is now making programming radically easier again.
With Unity + Cursor you can already build many things faster than in gamma simply because AI is tightly integrated into the coding workflow.

So if AI is democratizing programming, vvvv needs a clear answer for how visual patching and AI work together, otherwise one of our core advantages gets weaker.

At the same time, vvvv has exactly the right foundations:

🔧 Realtime C# compilation

🔧 A powerful visual node editor

🔧 Introspectable, strongly-typed patch structure

🔧 GPU-centric workflows (Fuse, Stride, compute-first)

If we add a proper AI layer on top – ideally via MCP servers – we can create something no traditional IDE offers:

➡️ A hybrid workflow: visual patching + generative code + AI-guided navigation + context-aware assistance directly in the patch.

This could bring in new users and unlock new creative workflows.

Next Steps

To move forward, we probably need to focus on two key layers:

A proper API surface for automation
Everything we can currently do manually in the editor (create nodes, connect pins, edit values, compile, run, inspect) should be exposed through an API.
This is essential so an MCP client (or any AI tool) can act on the patch graph safely and systematically.

Opening this API to the community
Once the API surface is in place, the community can experiment with:

MCP servers for vvvv

custom AI agents

alternative interaction models

patch generation/patch editing helpers

documentation bots

auto-fuse-code generators
…and explore what a truly useful AI integration should feel like.

So the path could be:

API → MCP/agent integration → community experimentation → best-practice UX emerges.
And yes this text is ai generated but based on previous thoughts and dictated audio

@david :
I agree to a 100% here. iorec and me had a talk regarding possibilities to get ZIM funding for such developments. I think the current state of vvvv and the timing regarding AI workflows is just right. I dont know about technical aspects and how complicated it is to create an API layer to start automate patching. But i would be keen to hear opinions for approaches here…

3 Likes
6

Here’s a clean, concise summary you can post directly in the forum:

Some members expressed concern about obviously LLM-generated long posts appearing in the forum or the chat. They feel that AI-styled writing — with generic enthusiasm, filler tone, or emoji-punctuated messages — dilutes the quality of human-to-human discussion. The wish is to keep conversations here authentic, thoughtful, and genuinely personal rather than bringing in AI-generated “slop.” Several users voiced agreement, emphasizing a preference for real conversations.

2 Likes
7

So before I begin here please let me apologize again for the ai generated translation of my thoughts that tebjan copied here it was a summary of ai chats and spoken thought transcribtion but in way to strong glossy llm vibe and totally in the oposite of what i want to communicate so this was a wrong move. I also understand the concern about AI-styled writing in the forum, and I share the wish to keep the discussion here grounded, personal and reflective.

That being said here are my thoughts on the topic

In the last two years, large language models have redefined how we work with code. Programming is moving from a technical and abstract discipline toward a more conceptual and creative one. If we understand vvvv not primarily as a “visual programming environment” but as a system for creative and conceptual programming, then this shift is directly relevant to us. The way we build and reason about systems is changing, and vvvv needs to respond accordingly.

At the same time, it is important to frame these models correctly. LLMs are not “artificial intelligence” in the sense in which this term is often used. They are a new medium — one that captures knowledge, thinking patterns, and the structures behind them. They allow us not only to consume information but to work inside a resonant space of knowledge. This is a substantial cultural development in its own right. It should not be reduced to AGI narratives that treat these models as autonomous entities and channel their output into the interests of a few companies. so my current frame of llms is not that of ai but of resonant knowledge fields.

As artist and creative technologist, I am deeply interested to keep a humanistic perspective on this development. Instead of reinforcing centralised feedback loops where a small number of actors shape the dominant modes of interaction, i am interested to build tools and workflows that let us intervene, think, experiment and create inside these new knowledge spaces. The goal is not passive consumption but active engagement: systems that support exploration rather than automation for its own sake.

From this perspective, the question for vvvv is how we position ourselves within this shift. vvvv already combines conceptual thinking, live coding, visual structure, and direct manipulation. If we integrate AI properly — not as an add-on, but as a way to navigate, search, modify and generate conceptual structures we enable the community to explore and shape the new possibilities llms bring.

One thought I’ve been circling around is that beside adding ai integration through mcp or similar solutions, we might need a simple way for a patch to describe itself — not to replace anything we do in vvvv, but to make its structure visible outside the editor. Something that just captures the essentials: what’s there, how it connects, which kinds of data flow through it.The idea is more about opening a small window into the graph so that both humans and language models can refer to the same thing when they talk about a patch.

The motivation is fairly straightforward. A patch is not text, and not code. It is a conceptual object with their own internal logic, and that logic is mostly invisible to a model right now. From experiment models can get patches already quiet well through screenshots but not communicate them back. If a model could see the actual structure — even in a reduced form — it could understand where it is in the system, what kind of operations make sense, and how things relate to each other. And it could do so without guessing or inventing structures that don’t exist in our environment and it could communicate back in the same language.

10 Likes
8

one of the most interesting topics at the moment 🍿

1 Like
9

I had this enlightenment last month of this summer, where I discovered the existence of local LLMs and MCP (Anthropic’s protocol) my first intuitive move was to see if it would be possible to integrate the VS copilot in vvvv (ambitious of course and far away from my knowledge basis). Some times although I get stubborn (most of the times tbh) and I tried to build my own custom vvvv-chat with OllamaSharp its tools and some MCP additions (last one didnt go really well since MCP is not yet stabilized and is changing day by day drastically). My humble goal was to create firstly a generic assistant to take care of my sh*t and mostly for github (there is already an MCP server for this by gh itself, however that time it wasn’t allowed to implement it in a custom client - VS and VS Code support it out of the box with copilot capabilities) .

This soon brought me to the next matter, on how I could use vibe coding (always with a small local LLM - like qwen3:8b) in the vvvv context. Again, I insisted doing and digging things that I am not fully understanding and I discovered that RAGs and embeddings would be the correct way to go.

My current state is a finished chat-client (always using imgui) in combination with OllamaSharp running models via DockerDesktop, a pretty small but “smart” enough model which is able to use tools and do basic agentic stuff (chaining), plus the urge to add external MCP tools (client-side).

However my obsession with this attempt, I agree that AI (as we know it in nowadays) is nothing more but a tool, a “sophisticated” statistical model, which can be pretty helpful in several cases, where in the same time can bring chaos and disrupt completely our information basis and paradigm, by introducing tones of garbage and nonsense stuff in the node based - human-centric programming.

But don’t get me wrong, I am not demonizing AI usage, quite the contrary, but I prefer to ground it rather than deifying it.

I am must sharing my thoughts.

2 Likes
10

qwen 8:b is kinda dumb, and terribly slow, I think the more or less ok 120 one, less then that it’s just for macroing also qwen not the best choice for c# at all sadly… I would say let’s check the copilot, check how we can provide context for it, (do we need to run own MCP server somewhere?) let’s start from adding simple help chat bot, then see if it can be pushed further… Likely there should be an api to patch programmatically first…

11

Question @joreg @Elias is there a way to use/download the graybook ? does it have copyright / infringement restrictions? Or this is summarizing in very simple way (I kept you like to simplify things) the need of a public API or of a REST API ?

12

@antokhio agree in every word, I am also exploring the options and capabilites, ended up with qwen because of its visual/image capabilities.

1 Like
13

Grasshopper is somewhat similar to vvvv - visual, node-based - they have just gotten the first AI plugin:

It’s actually open-source and written in C#, so maybe it could at least serve as inspiration for how to tackle this in vvvv.

From my limited research and experience it seems that the API would really be the biggest and hardest part. 2nd would be abstracting patches into something like JSON with simplified syntax (compared to a .vl file).

Actually that simplified JSON could even be used for a nice visual representation of a patch, which would be an awesome way to browse large patches.

Lastly, I do like the parts where you can just search the forum and documentiation and get a summary. That could be a nice little first step, that requires much less effort (no API needed). It’s similar to what you get in the new Affinity application. Its a small window to ask specific questions. That would already be helpful for simple “How do I connect X to Y” or “What node can I use to convert from A to Z?”. This would really require a written short explanation of every node, which I think would be nice anyway. Incidentally there is already something like this for Grasshopper: Grasshopper Component Index | Grasshopper Docs I think something like this always seemed like a bit of overkill in the past, but now that we have capable LLMs it turns out it was a really good decision to have something like that.

3 Likes
14

Here is one approach from cloudflare Code Mode: the better way to use MCP

  • LLMs have been trained on huge amounts of real TypeScript/JS/C#
  • They’ve seen very little synthetic tool-calling JSON / MCP examples

So:

  • models are surprisingly good at writing code against a normal API
  • the same model can be quite clumsy when juggling lots of small MCP tools

That leads to a different design pattern:

Instead of exposing dozens of tiny tools, expose a small, typed API and let the model write code that uses it.

For vvvv, this would mean:

  • don’t give the model a tool for every atomic operation in the editor
    (CreateNode, ConnectPin, SetPinValue, DeleteLink, …)
  • instead, define a thin “vvvv automation API” that feels like a normal library, e.g.:
    • OpenPatch(path)
    • GetGraphSummary()
    • FindNodeByTag(...)
    • ApplyPatchEdit(editDescription)
    • SavePatch()

Then let the LLM write code that:

  • reads a patch description
  • decides which edits to do
  • calls these few functions
  • and only returns the result back to the chat
  • probably executes the script with the api calls
3 Likes
15

This is an amazing discussion and I am really happy the community is having it :)

@seltzdesign Re the simplified JSON structure, I’ve had pretty good experience testing with YAML on a tree like structure for Schema - with basic instructions and node descriptions, ChatGPT could generate reasonable patches.

image
image1458×1844 264 KB

I agree with @texone above that the interaction layer should be a little higher level than placing nodes on a canvas.

I consider all layout to be superfluous to the LLM, it has to focus on the concepts and logical connections. The patch can be represented as an actual graph structure and laying out the nodes can be its own algorithmic process (force, hierarchical or other graph layouts).

I think that another question will be training data, we’ll need to find a way how to correctly annotate code blocks with meaning - comment regions or something even better.

2 Likes
16

I’ve been doing a lot of neural-network-assisted programming lately. Quite often it lets me ship whole projects without writing a single line of code myself. Right now I’m working a lot with Cursor. Everything you’re discussing here feels very important and very interesting.

I already see that I have to add some RAG-like rules for *.vl files in Cursor so it doesn’t pull them into the context. BUT sometimes it still does, and then it suddenly gives insights like: “you already have these nodes connected like this and that”. That’s pretty mind-blowing. It would be great if Cursor could skip raw *.vl files and all the “sugar” / extra noise, and include in the context only the important parts: the relationships between nodes, their structure, inputs and outputs, help texts on the nodes, for example.

But what I actually want to point out is that I already raised a topic that is already key for programming with neural networks:

You can make vvvv much more accessible for neural-network-driven workflows right now just by allowing seamless C# code execution — neural networks constantly generate code for vvvv that can be run with a simple copy-paste, but importing this code from AI into vvvv is a real nightmare at the moment. And thanks to this forum in particular, the models are already quite good at understanding many of vvvv’s limitations and nuances. How about starting not with an MCP server, but with making it easier for AI to drop C# code straight into a vvvv project?

MCP and RAG would definitely make the whole vibe-coding experience with vvvv MUCH better. I’m sure I don’t even need to “throw ideas” here about how exactly this experience could be improved (like reading the structure of *.vl files without overloading the context with garbage, or having a proper database of how different things are built and how they work). I just want to draw some attention to what can already be improved today, please. The last thing I personally need is an MCP server that patches nodes for me that I can already patch myself.

4 Likes
Experimental: Embedded C# Editor in Gamma HDE
17

@yar gamma has a pretty good support for C# projects with live reloading support.

The UX could be improved with an extension, but generating functional blocks (nodes) unlike full patches, was achievable even 1y ago

Exposing to MCP servers/vvvv specific RAG or extended guidelines would definitely make the outputs more grounded.

2 Likes
18

@domj, thanks. If you reread, you will understand that I am well aware that C# projects can be used.

The problem is that these are projects! They are not separate ‘scripts’ or files. Integrating code into Gamma is currently difficult, uncomfortable, and unnatural. It requires Visual Studio and a lot of knowledge, which is off-putting on many levels. One error in the project code and the entire project will not compile, regardless of the amount of code. Creating a project is a separate pain (imperceptible to an experienced developer, but unbearable when you don’t want to overcomplicate things).

I see the approach as follows: there are a bunch of *.cs files where I can write using neural networks and Cursor. I can quickly create a tool for myself without creating a project or a file within a project — that’s what I want. From idea to working node, there should be one prompt (not only in Cursor agents) and one code copy.

I’m also asking for the ability to code in place, not just the ability to connect pure *.cs files. I’m probably not too happy about the fact that C# code execution is supported, but it’s not possible to do so conveniently. There have been so many cases where I’ve received code from AI, but struggled to implement it.

1 Like
19

This is somewhat related to the topic: MCP extension for Max/Msp that got recently forked (and updated) by ersatz_ben: https://www.youtube.com/watch?v=ocUqYyPnfG4

20

I just want to give some feedback on coding C# nodes for vvvv using Cursor and Claude 4.5. I created a new C# file from within vvvv and it placed a node straight away, which I could then edit in cursor. I used the “Plan” chat to first explain what I needed with a few references and it then built the code for it pretty much one-shot. It needed a few iterations for details, but otherwise worked pretty flawlessly. I think I gave it some references to the grey book on Records. Claude 4.5 understood it was for vvvv and knew exactly what to do. It will actually trigger the recompile when it needs to, which was a nice touch. I sometimes had to restart vvvv, because it was somehow caching the .dll, but other than that it was pretty eficient.

The node I built, actually builds the XML of another .vl file. I needed it to read a configuration from CSV and create a lot of nested datatypes from it. I built a small sample file .vl by hand and had it analyise first and it “understood” what it had to do straight away.

Of course this is not the same as actually having an AI build patches, but I was still surprised how efficiently it could build the code I need in just a few prompts. Its a type of node that of course I could build in gamma itself, but it would be just some tedious patch with lots of ifs and switches just for building XML, which wouldn’t be very exciting and only really has one outcome - to create the file I need in the format I need it. I think for this type of small blocks of code using C# nodes built by AI is actually quite handy and was pretty straightforward.

I also believe that vvvv is actually an amazing platform to combine our own logic with building blocks from AI for specific things. Its good to understand and build the logic, but there are many times where you have to create a sub-patch just to do some tedious conversion and helper functions. If AI can build those in 1 or 2 prompts instead of me spending an hour on it, thats pretty handy.

Maybe we could start with some LLM friendly documentation. I was pretty impressed by the one ShadCN components have, which I have also used in another project: https://www.shadcn-svelte.com/llms.txt Its just a very condensed text file with a lot of the important information linked as .md files. Having something like that for vvvv would be great.

4 Likes