I subscribe to NightWatch, a nightly review of daily geopolitical events that is considered the best of breed. The interface is old school – just rich format email daily, and a web link that provides a list of former entries. No topic cloud, no search, no nothing – you have to pay attention on a daily basis.
I have long wanted to dig deeper, indexing the content, and today I struck out in a first experiment in this area. I wrote a python script to create URLs for each of the 156 watches published this year, imported them into Maltego, then ran the Alchemy and OpenCalais Named Entity Recognition transform.
There were a lot of singletons in the return and I guessed that many of them would be parsing errors, so I used the circular layout in order to select nodes with two or more links to move to a new graph.
Once I had the graph trimmed I examined the layout using ‘bubble view’. This looked promising, with nodes sized by degree and some underlying structure evident in the giant component.
But once I looked closely I was disappointed. These three section of the graph give you the idea that the system did a good job of sorting by region, but this is not the case – ‘Pakistan’ is on one side of the graph, while ‘Quetta’, perhaps the most violent city in the country, was on the other side.
This is a case of using the wrong tool to create something visually interesting, but that provides no insight. It’s useful to know how often topics get referenced, but when logically related items are spread all over the graph even that bit of aggregation of information is of dubious value.
This graph was created by accessing 156 URLs, each of which contains a date stamp. I could try the temporal analysis features in Sentinel Visualizer, but since we are trying to see concepts rather than forensic data, I am not sure that it’s the right solution. I probably need to sort out Gephi’s Graph Streaming plugin, but before I do that I’ll need to either extract the graph from Maltego or write something of my own to extract named entities from the NightWatch URLs.