ContributionManager for vvvv beta

[joreg]

Here goes, the never really promised, but long-planned contribution manager for vvvv beta:

Features

• lists packs in categories managed in a public repository everyone can contribute to
• prominently features the packs license (free, donation, commercial)
• provides a direct link to examples for installed packs
• packs can have different versions to choose between
• shows is a pack has an update available
• displays a packs (missing) dependencies so that you can install those as well
• allows you to add custom pack sources. local, as well as online ones
• automatically writes version.info for all packs, which allows the pack-maintainer to do future node renames
• has a global zip-cache, so that it doesn’t re-download packs all the time
• allows choosing between different “target” vvvv installations

Bugs

• we’re aware of random unfortunate freeze/crashes
• images are sometimes not showing
• when the list of packs is full (scrollbar showing) the last entry is not visible

How does it compare to vpm?

Apart from UI/UX, it seems the main difference is in the way pack dependencíes are handled: While vpm installs all dependencies of a pack automatically, the ContributionManager merely shows you a list of missing dependencies and needs you to install the missing packs. This requires a few more clicks, but gives you a chance to detect potential dependency conflicts.
Ah and vpm allows for more installation tricks, while ContributionManager only knows .zip files. And for vpm you can conveniently upload new packs via the vpdb, where the ContributionManager needs a pullrequest. Otherwise I’d say the ideas are pretty similar…

Will it be included with vvvv beta?

That is the plan.

Does it need a specific vvvv beta version?

No.

How does it work?

First press the “Configure” button to choose a vvvversion to install the packs to. The list you see there is all comprised of all vvvversions you installed using the installer plus, the first one is always the one that currently has .v4p registered.

If you want it to also show you vvvversions you installed manually, you’ll have to add those manually in the settings file:

\AppData\Local\vvvv\ContributionManagerSettings.xml

Then choose among the available packs to install. If after installation of a pack it shows a red sign labeled “Packs Missing…”, click it. This will show you missing dependencies of this pack and their minimum required version. Find those in the list and also install them.

When done, run vvvv and patch yourself away…

Click here if you're interested in technical details...

When starting it for the first time it creates a settings file:

\AppData\Local\vvvv\ContributionManagerSettings.xml

It allows you to specify local pack sources in addition to the default one which points to here:

https://github.com/vvvv/PublicContent/tree/master/ContributionManager/PackCatalog.xml

PackCatalog.xml is the main entry point. The catalog points to multiple category files. Each category file lists a bunch of packs.

A pack has a unique id, a human-readable title, author, info, image, website, changelog and license. License can be one of “free”, “donation” or “commercial” with a url for details.

Then a pack can have multiple releases: Each release has a version and either one item with cpu set to “any” or up to two items with cpu set to “x86” and or “x64”. Further, a release can specify a “minimum vvvv beta version” that it needs to run and a list of dependencies. Dependencies are the “Id” of another pack and a “minimum version” of that pack it needs to run.

Regarding pack Id: This must be a unique, case-sensitive string. For existing packs that already had a version.info and a diffff.xml, the Id must be the packs installation directory name.

Status

The list of packs currently provided is a random selection for testing. Please feel free to add your own packs locally for your own tests and make pullrequests on the repository to add more.

Any thoughts on the licensing options (free, donation, commercial)? Does it need more?

Changelog

v0.2

• now shows when a pack has updates available
• fixed: used to show wrong release on startup if not most recent release was installed
• fixed: last entry was cut off
• fixed: used to show vvvversion as installed if only path was available. now also checks for existing vvvv.exe
• fixed: got rid of some freeze/crashes. still some left it seems

Download

ContributionManager v0.2

Hope this works for you. Please test and report your findings!

[microdee]

cool!

The following is my constructive opinion/suggestions and not a rant, please imagine a kind soothing voice reading it:

Well you could have just used vpm as a backend to the UI, instead of reinventing the wheel. Or if you didn’t like vpm, you could have told me your problems/wishes, could have submitted issues or even better pull requests to the github repo, and come up together with a consensus, how actual open-source development works.

That aside the reason I used scripting in vpm was that many packs come in different folder structure or different installation instructions which can be scripted easily. Or via scripting for example one can get latest version of a pack from github releases, or latest revision of a minor version like 3.1.x, and stuff like that. mp.* packs for example are heavily relying on said scripts. Not just for getting the latest version but for example those packs rely on some C# libraries not shipped in the pack’s zip (to avoid attempting to load different versions of the same assembly) so they either separately need to be placed in “lib\core” vvvv folder, or edit vvvv.exe.config to probe a folder in the packs folder too for dll’s. With Contribution Manager I can not do this apparently. mp.* packs cannot be distributed in a civilized manner with it, especially when they have dependencies which in ideal scenario the user can completely ignore (I’m talking about md.stdl, mp.pddn and Notui in this particular case).

Just to demonstrate my Notuiv or Vanadium packs have this dependency tree:

• Notuiv
  • Notui (with vvvv.exe.config modification)
  • [shared deps]
• mp.dx
  • dx11-vvvv
  • mp.fxh
  • mp.essentials
    • Messages
    • md.stdl (with vvvv.exe.config modification)
    • mp.pddn (with vvvv.exe.config modification)

Good luck manually going through them, imagine the tedium :P

However I was a complete ignorant, prejudicial idiot when I chose C# for the scripting and XML for the metadata. Scripting with Powershell would have made so much more sense, especially considering that

• A: PS is a .NET language, operates with .NET objects, so what you can do in C#, you can also do in PS (oversimplified ofc); and
• B: PS is waaay better suited for file system operations/sysadmining like package management than the general purpose (and for this purpose waay over-powered) C#.

And then one could have embedded the metadata in a multiline comment in the Powershell script/module representing the package.

When I presented the initial coding drafts for the supposed successor to vpm at LINK (uppm), I was still an ignorant idiot. If I would come back to its development I would ditch C# scripting completely, ditch the own standalone console layer, and have a layer of Powershell module instead over the package management classes. You can then build a UI either on top of the Powershell module or use the class library directly. The name resolution, the target application space management and the dependency tree generation would still be necessary and be part of the C# class library of the package manager.

Now I actually have time to get back onto this bandwagon and continue uppm development where I left off, it has really good potentials and the code in there is already solving some really hard caveats of proper package management (like version requirements and therefore dependency version collision detection). When it becomes usable it is also a completely manageable task to convert the vpdb catalog into the uppm format.

[readme]

@microdee

Or if you didn’t like vpm, you could have told me your problems/wishes, could have submitted issues or even better pull requests to the github repo, and come up together with a consensus, how actual open-source development works.

I have to agree here and in fact am guilty of these accusations as well. I never really filed actual feedback of what didn’t work/feel right for me, but joining forces really seems to make sense here without shutting (without any bad intent) a user like David out.
But this is a problem that doesn’t apply to this case alone, somehow we as a community often manage to reinvent the wheel a couple times - often without trying to identify or debate about shortcomings of some side-projects (timeline, ui libraries, etc.).
For some things it’s not easily done, but I feel as if this package manager could benefit from David’s experience.

[joreg]

@microdee: of course vpm would have been an option. here are our considerations for why we didn’t opt for it: ultimately package-management for us is solved with nuget. nuget will be the main backend for the vvvv gamma manager. nuget3 mostly removed scripting because in their experience it caused too many problems. if a pack needed a script to be installed they argue they rather solve that issue than allowing scripting that can get unmanageable. we don’t want to reinvent the wheel.

only in addition to nuget, as a simpler to use alternative, we wanna offer the simple most possible solution for people who cannot use nuget. this is what we were aiming for with ContributionManager and its “backend”: just meta-info and a zip-file, nothing more. so for what we want it to do, it would be overkill to use an external solution like vpm.

also a tool like the contributionmanager for us makes a perfect case for dogfooding. the goal was to implement the whole thing using vl: download and visualize meta-info, then download and unzip files. so for us it is not only about creating a contributionmanager but also the fact that it makes for a perfect proof of concept and test-case for what you can do with vvvv gamma.

@microdee

many packs come in different folder structure or different installation instructions

they really shouldn’t! we have a specification of how a pack must be structured to be valid, see: HowTo Release a Pack. and these are the packs we support with ContributionManager. i’m quite certain for most packs currently not conforming to that spec a simple re-upload (with fixed dir-structure in the zip) will make them work. no scripting needed.

do you know of any other than your own packs that don’t (or cannot) follow those specs?

@microdee

they either separately need to be placed in “lib\core” vvvv folder, or edit vvvv.exe.config to probe a folder in the packs folder too for dll’s

are you sure these are the only solutions? it seems your packs are modifying vvvv.exe.config to add a custom searchpath to the \md.core pack. shouldn’t it be enough to make this one pack follow our specs and put all the shared .dlls into \md.core\core in which case they would already be picked up?

[microdee]

Oh maan, honestly during all these years, I’ve not realized that the managed DLL dependencies would be picked up in the pack subfolder core, stupid me, I will try that then because that’s way much more elegant than XML editing of the vvvv.exe.config.

@joreg

do you know of any other than your own packs that don’t (or cannot) follow those specs?

Well not out of my head. Afaik no one else took really full advantage of it, many people just copied my example scripts and changed the names. I wrote installer scripts for vvvv itself way before the actual installer came. That’s the only legitimate thing which essentially require scripting, and it’s not even a pack. I use the scripts to get the latest releases from github and do some other file juggling, so I can just use automated releases, sparing me manual uploads every time I’m adding a little feature or fix a bug. I actually wanted to have ambitions with license checking commercial packs or configurable installations which were all there but that was dead at arrival, too elaborate I guess to catch on.

Don’t get me wrong, the choice of Nuget for VL and gamma comes naturally, I’m very happy that you decided on Nuget there. We’re not talking about them. We’re talking about legacy old-school grandpa vvvv which unfortunately is not good with, and not using Nuget (again old-vvvv without VL) so in this very case, you did reinvent the wheel.

Nuget is a very strictly defined system with a lot of infrastructure and tooling for a very specific purpose. It makes sense there to encourage as little scripting as possible. You simply cannot participate in that ecosystem if you’re not following strict .NET development specific folder structuring rules, a very well defined metadata description and an archive format. On the other hand though because vvvv packs historically meant to be installed manually, and because there’s no “physical” enforcing of the “packs format” which came around just before dx11-vvvv came out afair, it’s the wild west out there regarding structure, (the existence of) metadata and delivery methods. I chose to have scripting in VPM because I didn’t wanted developers to change their habbits (and I assumed they will not come on board if they have to). Also I wanted to make existing packs part of the ecosystem as smooth as possible, without the need for repackaging, or including still widely used packs which has been abandoned by their original creator khmcvimagepackkhm. That is debatable how it turned out, but I’m just a single person doing that btw open-source project, next to many others I’m trying to find time maintaining next to my day job, which has exponentially less to do with vvvv.

Notice unfortunately there’s no versioning in VPM. That’s because I can count the amount of packs which has version.info on one hand, let alone diffff.xml. Sadly very few (= 1or 3) people provide that in the packs. I’m also guilty of only generating the version.info on the first runtime of mp.essentials, mp.dx and Notuiv, but not shipping with one (and not even having it in my other packs). How on earth one would have done versioning with that culture? (also I was significantly worse at coding 4-5 years ago, today I have actually a draft for a solution for that in uppm)

One more thing on the folders: Well yeah there’s the pack folder convention, but then contribs (many of them being wide-spread) doesn’t follow that consistently (regarding which folder level is the root of the archive of the pack) . My packs for example assume the “packs” folder as root, but DX11’s archive root is the vvvv folder not the “packs” folder. I’ve seen some packs where the archive root was the pack folder inside the vvvv/packs folder (so in manual installation it was the user’s responsibility to create the folder for the pack inside the vvvv/packs folder). Some other contribs are just a single or couple DLL, effect or v4p file(s).

Eh so you see my point on scripting. Very sadly the contribution culture of vvvv while very vibrant and pretty decent in size imo, is not at all consistent with format. I wouldn’t bet many people will update their packs to participate in this, but maybe that’s just my grumpy angry old man attitude.

[joreg]

@microdee

I will try that then because that’s way much more elegant than XML editing of the vvvv.exe.config.

good, let us know if there are any further issues.

regarding version.info:
forgot to mention above: ContributionManager writes a correct version.info after unzipping a pack. next to the version, this file now also includes a packs unique Id (mentioned above). this allows us to provide a simple form of versioning where we can display dependencies with their min-version and display their status (installed=blue or missing=red).

regarding pack structure:
the ContributionManager can actually handle 3 different types of folder structure in zip files to cover most cases that we see on Contributions atm:

• root is a \packs dir
• root is a \packname dir (this is what most packs do already)
• root contains neither \packs nor \packname directly
  many other contributions indeed do not follow any of those but anyone can re-upload those easily and thus help making them available via the ContributionManager.

[tgd]

just realized this a VL app lol. curious to peek inside. can we have a public repo for this at some point?

[joreg]

@tgd

at some point

of course.

[velcrome]

@joreg

at some point

of course.

when is this scheduled? or rather, what stops you from having it open now?
tbh I don’t really get the reasoning of having a wip peer review without any source, especially when you know it is buggy

@joreg

do you know of any other than your own packs that don’t (or cannot) follow those specs?

just from the top of my head: dx11, vvvv-Message and ui-Automata

in my case it is not a “simple” upload, because I automated pack building with appveyor and vpm years ago. really glad your Manager is capable to handle that anyway.

my biggest concern right now is your decision to reference multiple contributions from multiple maintainers in one file in your github repo - that is bound to cause merge trouble as soon as someone uses different line endings, similar to what happened with the addonpack.csproj back when there was still activity.

what I am missing too:

• User acknowledgement for non-free licenses before install
• remove/uninstall option
• price, if there is any (not just a link)
• max version indicator (in case a breaking change happens in a dependency, that hasn’t been taken care of)
• batch support, so a project can have an executable list of all necessary packs

[microdee]

@velcrome

in my case it is not a “simple” upload, because I automated pack building with appveyor and vpm years ago.

Of course same goes for my packs, doing appveyor and github releases. But tbh in the end of the automated build one can create a pull request for the package repo and at that point you don’t need scripting. (well on the client side at least, you’d need some decently elaborate batch/powershell script on the server) (or use a webapp you set up so one does not need to merge pull requests manually). I agree that the single XML file holding many packages is a really stupid design decision. Why do that? Why not a single file per package? And on that note why XML? Why not YAML or HJSON?

[Hadasi]

@microdee

And on that note why XML? Why not YAML or HJSON?

Dog fooding. There isn’t a deserializer for those filetypes in VL yet. Its a work-in-progress, so all this feedback will be welcome. I don’t understand: is the issue with the file with the fact its static? The code for the ContributionManager itself has a separate repo, so updating either won’t effect the other.

I suppose some sort of system for keeping track of all the contributions will be needed, but I wonder if that effort would land on the shoulders of the ContributionManager manager or the contributor themselves. I guess your approach was to spin up a server and do it all via a web app. I’d probably look for a simpler solution first that might mean a little more work on the contributor’s side cos I’m allergic to web.

@velcrome

when is this scheduled? or rather, what stops you from having it open now?
tbh I don’t really get the reasoning of having a wip peer review without any source, especially when you know it is buggy

I see this as more of a design review rather than a code review, comparable to how we give feedback about Gamma/VL itself. It did crash on me however, (press ‘Details’ for ParaLib, it hung and it crashed without warning) and I can’t give much more technical feedback on it than that.

I think the ContributionManager could do with more space for notes as @velcrome suggested, Especially regarding licensing.

Looks pretty good, but the bottom cuts out.

[microdee]

@Hadasi

It did crash on me however, (press ‘Details’ for ParaLib, it hung and it crashed without warning) and I can’t give much more technical feedback on it than that.

You should look into Windows Event Viewer when that happens and see if an application error has been logged (even better if from .NET) because the contents of that error event will actually give you a managed exception with a message and stack-trace, which then gives more information to devvvvs.

On XML and dog-fooding: yeah right VL has its serializer. I despise this dog-fooding argument 1st: is this a sample application or a component which you take seriously? If the latter then this project should sprout new tools/technologies*, not just look at what is in the fridge and then throw together something quickly. 2nd: In fact, I feel very butthurt from that argument, because that means devvvvs chose a quick VL sample application for managing packs for their entire legacy software line**, developed in tops 2-3 weeks, instead of a well tested and widely used solution, with an already existing catalog. Meanwhile being so ridiculously fucking over the top careful about making DX11 vanilla, and then deciding against it for some pretentious made up reasons. This just leaves a really bad taste in my mouth.

* like for example make that serializer output format “agnostic”
** which btw they still supposedly maintain, and not announced end-of-life yet

Anyway, I will shut down vpdb and make the vpm repo private in march, in favor of whatever this garbage is. If anybody wants it still around they can fork both vpdb and vpm, and make their own frontend for vpdb. This also means my packs no longer will be easily installable for a while after march, I might cook up some powershell scripts for that, but might not happen.

[Hadasi]

@microdee

On XML and dog-fooding: yeah right VL has its serializer.

Maybe I meant parser. It doesn’t have a parser for yaml or Hjson as far as I know, but it could of course be implemented. I know some of the drawbacks of XML over yaml and json, but its a detail. I think the broader detail I was getting at - static file vs active server - was the genuine system design question I was asking.

I understand your frustration at the fact they put this idea into action without discussing it with you given all the work you’ve done in the area, at the very least from a design point of view, and considering code too. This is a point I think I’ll expand upon later but in short its a matter of trust and appreciation. I think your point about dx11 is merited, but maybe digresses from this specific subject.

You may have heard this sort of thing before but the problem with vpm was that I tried it a couple of times, and it kinda worked and then kinda didn’t, and then I moved on. Its jaw-droppingly clever, but it didn’t work i just thought I know a way to do load packages without solving this puzzle or writing a bug report. I probably should have done. Sorry dude; its disrepectful to your work to not give you feedback about it as a friend to make it better, especially since its free. @microdee, you lovely, furry, gold-quad bastard, you xx. Honestly, dude, you’re the man.

But I’m going to give @joreg credit here. He’s previously been overly-cautious about actually implementing the things that they think and talk about ad nauseam in meetings. These ideas stew for yonks including this one, thinking about your work that they have a lot of admiration for. I give him credit because he’s taking a punt to actually implement something they know is needed and to see how we feel about it.

I think the reason for the decision to start from scratch is part of the recent push to deal with the accessibility of vvvv as a whole, which you can see in the help browser of Gamma and the help stuff they dropped just now. It may not seem like it but I could imagine new users make the same mistakes that I made. I’ve seen people give up on software for much less, and I suppose they’ll want to incorporate both Beta and Gamma contributions into one manager too.

And, of course its also a matter of control. Their design aesthetic, their program patterns, their hexagonal wheel made there. But we all have a bit of that.

I hope they continue this experimental approach with the main products too so we could see faster iterations and change in areas we think VL can improve but which haven’t escaped the *kitchen.

But briefly on that other thing:

@readme

But this is a problem that doesn’t apply to this case alone, somehow we as a community often manage to reinvent the wheel a couple times - often without trying to identify or debate about shortcomings of some side-projects (timeline, ui libraries, etc.).
For some things it’s not easily done, but I feel as if this package manager could benefit from David’s experience.

I whole-heatedly agree. The back and forth between the vvvv group and the community devs has been strained over the years because of lack of feedback on all sides. As a community we have to accept that sometimes the vvvv make choices that are in the interest of the company in ways we can’t see (time, design, maintainability). But I hope they will turn to the community as a resource more often. I guess the difficulty is in how these relationships and collaborations are managed and nourished.

*meeting room

[joreg]

@velcrome: as @hadasi pointed out: at this point we’re interested in feedback on the functionality not a code review

@velcrome

just from the top of my head: dx11, vvvv-Message and ui-Automata

from what i can see those 3 packs follow the specs and therefore are working. why do you think they wouldn’t?

regarding the choice of xml and the particular file structure:
i don’t want to defend this too much other than saying that we thought about what would be the simplest approach to get a working result that is easy to maintain manually. ultimately ideally the xml would not be visible to anyone and we can think of a nice ui-frontend and web-api to upload meta-data to. but then again that already sounds a lot like what nuget has…

@velcrome

User acknowledgement for non-free licenses before install

worth a thought. only what i don’t like about this, is that this also adds this annoyance to click twice for those who do pay. every time. also consider batch-support (see below) will not like this. so let’s see if there is more feedback on this…anyone?

@velcrome

remove/uninstall option

you mean of ContributionManager itself? the packs can already be uninstalled, right?

@velcrome

price, if there is any (not just a link)

good point, but how exactly? if you look at the store atm, some packs are licensed per PC others per project and HAP has two different prices… so showing a single price wouldn’t suffice. only the link allows to express custom pricing in all detail.

@velcrome

batch support, so a project can have an executable list of all necessary packs

yes, it is planned to be able to export the current set of installed contributions and then automate their installation.

@microdee that escalated quickly: we’re aware that you’re critical of a lot of what we do since a while now. we always value your input, please keep it coming, but also please don’t expect us to simply jump to your conclusions. at some points i’m afraid we have to agree to disagree:

@microdee

is this a sample application or a component which you take seriously? If the latter then this project should sprout new tools/technologies

our thinking: it is exactly both and therefore deliberately does not sprout new tech: our sprout is vl as a whole. we want to show what it can do right now and build something useful that anyone eventually can also learn from. we did the same with the tooltips and helpbrowser and are planning to do more and more in that direction going forward.

taking down vpm as a reaction to us trying to create an alternative solution would be a great loss for all people relying on it and vvvv as a whole. you built vpm when we didn’t yet have it on high priority and we’re grateful for that just as we are for any contribution that improves the vvvv ecosystem.

@hadasi thanks for your sensitive analysis of the matter. nothing to add at this point.
and the problem with the cut-off last entry is now fixed, please try again.

[microdee]

@joreg

yes, it is planned to be able to export the current set of installed contributions and then automate their installation.

I’m very happy to hear that actually.

Sorry about my tantrum, I will give that shutdown more thought, it might be overkill you’re right.

[motzi]

@joreg

only in addition to nuget, as a simpler to use alternative, we wanna offer the simple most possible solution for people who cannot use nuget.

having a simple (i.e. GUI) option to install certain VL nugets would be nice to have. the use case would be a pack containing vvvv-wrapped VL nodes that depend on a nuget (e.g. from VL.OpenCV).

@microdee: is installing a nuget to lib/nugets possible with vpm easily (I guess it should be possible using a C# script calling nuget.exe)…

[microdee]

@motzi

is installing a nuget to lib/nugets possible with vpm easily

yup, but why would you from vpm?

[motzi]

@microdee

yup, but why would you from vpm?

simply because installing a package and automatically get all dependencies resolved and installed as well in one go makes life so much easier ™
VL.OpenCV would be such a dependency for a package I’m preparing.

could you give a hint how to do that?

[microdee]

@motzi in the scripts section paste:

using System.Diagnostics
...
var nugetProc = Process.Start(new ProcessStartInfo
{
    FileName = $"{VVVV.Dir}\\lib\\packs\\nuget.exe",
    Arguments = "install VL.OpenCV -prerelease",
    UseShellExecute = false,
    WorkingDirectory = $"{VVVV.Dir}\\lib\\packs"
});
nugetProc.WaitForExit();

[motzi]

thanks man!