Illuminating NightWatch

I subscribe to NightWatch, a nightly review of daily geopolitical events that is considered the best of breed. The interface is old school – just rich format email daily, and a web link that provides a list of former entries. No topic cloud, no search, no nothing – you have to pay attention on a daily basis.

I have long wanted to dig deeper, indexing the content, and today I struck out in a first experiment in this area. I wrote a python script to create URLs for each of the 156 watches published this year, imported them into Maltego, then ran the Alchemy and OpenCalais Named Entity Recognition transform.

NightWatch Named Entity Recognition

NightWatch Named Entity Recognition

There were a lot of singletons in the return and I guessed that many of them would be parsing errors, so I used the circular layout in order to select nodes with two or more links to move to a new graph.

NightWatch NER Circular Layout

NightWatch NER Circular Layout

Once I had the graph trimmed I examined the layout using ‘bubble view’. This looked promising, with nodes sized by degree and some underlying structure evident in the giant component.

NightWatch NER Entities

NightWatch NER Entities

But once I looked closely I was disappointed. These three section of the graph give you the idea that the system did a good job of sorting by region, but this is not the case – ‘Pakistan’ is on one side of the graph, while ‘Quetta’, perhaps the most violent city in the country, was on the other side.

NightWatch NER Topics #1

NightWatch NER Topics #1

NightWatch NER Topics #2

NightWatch NER Topics #2

NightWatch Topics #3

NightWatch Topics #3

This is a case of using the wrong tool to create something visually interesting, but that provides no insight. It’s useful to know how often topics get referenced, but when logically related items are spread all over the graph even that bit of aggregation of information is of dubious value.

This graph was created by accessing 156 URLs, each of which contains a date stamp. I could try the temporal analysis features in Sentinel Visualizer, but since we are trying to see concepts rather than forensic data, I am not sure that it’s the right solution. I probably need to sort out Gephi’s Graph Streaming plugin, but before I do that I’ll need to either extract the graph from Maltego or write something of my own to extract named entities from the NightWatch URLs.

Advertisements

2 thoughts on “Illuminating NightWatch

  1. Pingback: Defining Fameball Decomposition | Neal Rauhauser

  2. Pingback: Defining Fameball Decomposition by @nrauhauser | Fameball Decomposition

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s