I have been seeing #TheResistance hashtag in conjunction with The Indivisible Movement so I decided to take a closer look.
Using FollowerWonk revealed 4,800 profiles using the hashtag, so I took the somewhat unusual step of paying for a month of the service so I could download them all. I set a single account in the Netwar System to record them, and this morning I had 738 complete profiles.
When I combined the friends files for this group I found an amazing 378,00+ nodes with 1.16m links. This is what I’ve got after a half an hour of OpenOrd attempting to lay out the network. I dedicated seven cores to the job, my fan sounds like a jet engine, and the entire body of my MacBook Pro is warm.
I sorted by degree and found there were 823 nodes with a degree of more than 100. Given the demands 15.3% of the data set have put on my system I’m curious to see if it will even import the whole thing.
It seems to be taking about 90 seconds per account to preserve them, so 4,800 accounts will take 7,200 minutes, or about five days. Once that’s done I’ve got an existing ELK server and a shiny new Kibi setup. I’m going to have a go at importing and indexing an estimated 15 million tweets. I don’t have a specific goal in mind with this, other than getting more familiar with what these two Elasticsearch systems can do.