IPv4 space data visualization

[fibo]

Hi all!

Probably you know what is an IP address, for example vvvv.org domain has IP address 148.251.126.19 where numbers can vary from 0 to 255.

There are few vvvv nodes related to this network topic, for example you can ping an IP address (there is a node for that, right?) and the server can respond or not.

Ok, now immagine to ping all IP addresses, I mean all, the whole vintage Internet IPv4 space, and to visualize somehow that information, for example a red dot if the server answers to the ping.

Yes, you can also scan for other informations, like open ports, if it has an FTP or HTTP service running, and other Nerd stuff, by now I am pinging all the IPv4 addresses in the holy Internet and visualizing it on a website, for example vvvv.org is one of the red squares in this picture https://ip-v4.space/?subnet=148.251.126.

It is a lot of data, saved in JSON format (I could export it also on CSV or something else) and it would be interesting to feed vvvv with that data to visualize it, maybe in 3d. I have done some experiment like this in the past, for example mapping the IPv4 space on a sphere and there is a trippy maps that in my opionion fit well, it is the Hopf map (yes, yet another German mathematician, I need to learn German to read good mathematics) but that is just a trip of mine.

I would like to share the data and see how you could visualize it.

I can help writing patches, nodes, or shaders (if I still remember how to do it :) but I prefer to keep the focus on the scanning, which requires some effort, right now it is around 85% completed (it is the whole IPv4 addresses of the Internet) and I would like to improve it, make it faster and add more informations like the country of the IP (which I have no idea how it could be visualized).

If you want to join or just want the data (for free, no interaction or exchange required), or you are curious and you would like to talks about it just for fun, of course we are going to meet at the Hacker space at node 2017, fucking yeah!

[velcrome]

the whole IPv4 space is much less dense than I would have expected. but even than 2 billion hosts are quite a bunch to visualize. but playing with that sounds really cool

I can envision particles, with interactive filtering and highlighting. I think the new ChunkReader could help get started, but that needs some importers to generate chunks.
What does your json look like? Can you show a bunch, for feasibility?

[soriak]

hi fibo,
what are you using to ping at the moment?
i wrote a node a while ago to do asynchronous pinging as quickly as I could… but i think i reached a limit of 200/s or something

[fibo]

@velcrome yes 2 billion hosts is a lot, think about actually print the whole internet LOL. I am using one second timeout so it is also 1 billion seconds computing, luckly it is possible to parallelize it.

The gren square is your position.

The JSON files are on a path like

data/$A/$A.$B.json

on ip-v4.space domain, where A and B are the first two digits of the IP address, so the JSON related to the link posted above is

http://ip-v4.space/data/148/148.251.json

and it contains an array of elements with the subnet and a ping attribute with 256 elements that cna be 0 or 1.

for example

{“ping”:[0,1,1,1,0,1,1,0,1,1,0,1,1,1,1,1,0,1,1,1,1,0,1,0,0,0,1,1,0,0,0,0,0,1,1,1,1,1,1,0,1,1,1,1,1,1,1,1,1,1,0,0,1,1,0,1,1,0,0,1,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,0,1,0,1,1,1,1,0,0,1,0,0,0,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,1,1,1,1,1,1,0,1,1,1,1,1,1,1,1,0,1,1,0,1,0,0,0,1,0,0,0,0,1,1,1,1,1,1,1,1,0,0,1,1,1,1,1,0,1,1,0,1,1,1,0,1,0,0,0,0,0,0,0,0,0,1,1,0,1,1,0,1,1,1,1,1,1,1,1,1,0,1,1,1,1,1,1,0,1,1,0,1,0],“subnet”:“148.251.0”}

@soriak I am using 4 Ubuntu servers that launch ~250 processes each one in parallel, everyone pinging a classC subnet, when there are less than 250 processes other can be launched.
Result files are uploaded on Amazon S3, and the process checks that the file does not exists yet on S3 folder, otherwise it stops and the next process can be spawn. It is almost 85% completed, when finished I will update the scripts to grab other informations that could be if it has an HTTP server, if it has a favicon and so on.

Current repository is here GitHub - fibo/netvision: IPv4 space data scanner

[velcrome]

that was good fun for an hour. was considering this as an exercise for json parsing for my workshop, and got carried away a little.

IPv4.v4p (108.7 KB)
needs Message, have fun

[fibo]

Ah great, you could parse JSON!

If it makes sense I could create a data.zip file containing all the JSON files so it would be possible to load them offline.

[velcrome]

another evening, another hour well spent.

this time it will parse to Particles. Still via Message, because I needed the throttling mechanism with Message, and after hustling with Counter, I stopped bothering with vanilla vvvv. I dare anyone to replace all Message nodes with VL ;)

The main focus this night is dx11-particles, though.
My GPU capped at visualizing 4,194,304 hosts simultaneously, but still had 25 fps. not bad, considering the pc I am using here in my living room is 7 years old, and the last upgrade ever a gtx 770.

I like the human aspect of discovery within raw data visualisation: much less pings seem to return then when you started, @fibo!

Patch:
IPv4_particles.v4p (71.1 KB)

[velcrome]

If you are missing dependencies to test above help patch, put VPM next to your vvvv install, run it once and give it admin rights as instructed in the console.
After that simply go here for dx11-Particles and go here for vvvv-Message

This will also prepare you for node17, so make sure to run vpm with a 35.7 (v4p files must be registered to it) to be up to date when you arrive.

[fibo]

Wow, it looks like a really great result.

I am thinking about map the points on a Torus, it would be great if we can meet in hacker space and tpatch together.

There is also a JSON file representing the whole internet, it is the first data I show when you go to the ip-v4.space homepage:

http://ip-v4.space/data/master_tile.json

[velcrome]

sounds good.

btw, what is this?

… 1,1,1,1,“-1”,1,1,1,1,1, …

it is from the master file, some ping arrays contain similar things.
what does -1 mean, and why is it a string? funkiness like that makes it harder to parse.

edit: cleaned up version.

IPv4_particles_masterjson.v4p (62.0 KB)

[tmp]

nice project :)

should be no problem to use the chunkfile logic from the particles pack to explore the data in realtime (see chunkstreaming example in girlpower folder). all we need is a good idea how to distribute the data in space.

[fibo]

@velcrome, -1 means “no data available yet”. It should be a number, it was a string due to a bug, now the master tile JSON file should be cleaned up.

There are whole areas, like 16.* addresses (all belong to IBM if I am not wrong) that do not respond.
Other areas where not scanned yet, it is around 85% completed.

@tmp: thank you very much, to distribute data in space, maybe we can use a Torus.

[velcrome]

so we need a mapping F like this

(X, Y, Z) = F(Subnet, Index)

Do you know how F could look like, if it were to be a Torus?
would it be easier, to define F with full internet in mind, like in this

(X, Y, Z) = F(SubnetA, SubnetB, SubnetC, Index)

any idea?

[fibo]

I have this old patch with th e Hopf map.

I do not have vvvv on this computer, I am on a Chromebook, so I can check it out in the hacker space if I find some vvvv instance available.

The Hopf map implemented here it is an Hopf fibration and sends a point in R4 (that is what we need, we have an IP address that has 4 coordinates) into a point of R3, so it should sort out the points in 3d.

Hopf.v4p (9.2 KB)

[velcrome]

thats a tad complex, at least to me. i have to trust your math on that one. I tried to find a meaningful mapping with your Hopf module, but somehow I cannot see a meaningful shape.

hopf.zip (37.5 KB)

The question at this point is: What is a good value range to map the subnet’s 0…255 to good quaternions, so the Hopf produces interesting shapes?

[fibo]

I am trying to run the patches, sorry it is a lot of time that I was not using vvvv, so I need time to get into it, for example I still could not get the vvvv-Message node working.

[h99]

@fibo Hi, really just a silly thing, but would you consider to add text-align:center; to your css within *{...}?
This would roughly center the content.
I’d have sent a mail via your profile, but there’s no mail button.

[fibo]

@h99 in which web page are you suggesting to edit CSS?

[h99]

@fibo Ahahah, I thought you were the admin of ip-v4.space, oh my gosh.

[fibo]

@h99 yes I am the admin of ip-v4.space website. It is supposed to be a mobile first website, if you resize the window it will adapt to portrait or landscape orientation.

The rectangle outside the ipv4 square, it is for adding controls and infos, by now there is only an “up” button.