Author Archives: Neal Rauhauser

About Neal Rauhauser

Itinerant knowledge engineer, relentless scribbler, amateur photographer, troll busting ninja.

GDELT’s Analysis Service

I first noticed the Global Database of Events, Language, and Tone on New Years day, which led me to write Foreign Policy’s Global Conversation Infograhic. Three weeks later the service was mysteriously suspended and five days later we got an explanation as to what happened.

Since the successful resolution of concerns regarding data and processes used by GDELT they have made dramatic progress. GDELT Analysis Service offers fourteen different visualizations and export of the associated data. I entered the keyword ‘Ukraine’, selected all data for 2014, and about a minute later I received an email with this link to a live heatmap and another link to a CSV file.

Ukraine Mentions During 2014

Ukraine Mentions During 2014

Next I used the timeline tool. I was hoping for a live page like the location heat map, but it just produces a static image and a CSV file.

Each GDELT event contains a date, geographic coordinates, the players involved, and the type of interaction. The content is regular enough that it could be mapped to an import process for Sentinel Visualizer, an intelligence sector link analysis tool. I had set out to do something similar with the Global Terrorism Database last summer, but there were license issues, and the dataset lacks the live feed feature that GDELT provides.

GDELT is clearly going to grow services based on the live stream of content they have available. These services are likely to play to the strengths they have in defining and operating the stream. There should be plenty of room for follow on qualitative analysis and integration of data sources external to their feed. Data sources such as spot commodity prices in locations trending towards trouble would be particularly helpful in spotting hazards before they get out of hand.

Global Database of Events, Language & Tone (GDELT) Is SAFE!

That was a long, uncomfortable silence, after I posted GDELT’s Mysterious Demise, but we now have the particulars on what happened:

The bottom line is that GDELT is one of the very few event datasets in existence today that actually has all of the necessary permissions. The concerns that have recently been discussed were raised by two faculty members at the University of Illinois and were examined by a panel of faculty experts convened by the University of Illinois’ Vice Chancellor for Research. That panel formally cleared GDELT on behalf of that office stating “the Panel finds that it was not able to conclude that GDELT is founded on misappropriated … data or software.” With respect to concerns raised regarding the open source TABARI software that GDELT makes use of to create its CAMEO event records, the same panel explored concerns raised regarding its ownership and similarly found that “TABARI … has well known antecedents at another institution dating back to at least 2000 and therefore is not attributable to the [University of Illinois]“. While this whole situation would have been easily avoided with just a little communication and avoided a lot of unnecessary angst, the silver lining is that it has demonstrated just how widely-used and important GDELT has really become over the past year and we are tremendously excited to work with all of you in 2014 to really explore the future of “big data” study of human society.

I thought there might be a problem with either the underlying data or the software used, turns out that both issues were raised by the University of Illinois professors who parted ways with the project.

This feels a bit like the USL vs. BSDi lawsuit, which freed unix from AT&T’s clutches twenty years ago. A big, important datasource is now out in the open in such a way that it can not be put back. I have some financial records digging to do in the coming week, the Montgomery County Council will remain a priority until the primary is over, but I am itching to wrestle the GDELT feed into some format I can personally use.

Geospatial Tools For Activists

When working with grassroots analysts free software and services are key. I have long wished for a project that would fund a copy of Sentinel Visualizer I could keep, but the $5,000 cost for being able to handle temporal and geospatial data is very steep.

I have been inspecting campaign finance data for the Montgomery County Council and the files are clean enough that all of it could be geocoded, the only question was what tool to use. Earlier today noticed Sourcemap in a logistics discussion group on LinkedIn and this just might be the solution we’ve been seeking. Here is an example map from the free service, showing the global sources of Nutella ingredients.

Nutella Sourcemap

Nutella Sourcemap

If you sign up for the service it provides you a way to do free, public mappings using a simple spreadsheet format for input.

Lemonade Spreadsheet

Lemonade Spreadsheet

We can treat financial contributions as raw materials, county council members as ‘legislation factories’, and someone would have to know enough to describe which developer goes with which specific project, then we’d have a finished map of influence to development.

I can do the data handling component, what I need are some local hands and eyes who know more of the companies, personalities, and history behind the urbanization of Montgomery County.

GDELT’s Mysterious Demise

I wrote Foreign Policy’s Global Conversation Infographic on New Year’s Day. The content used to create the visualization was based on the Global Data of Events, Language and Tone, commonly referred to as GDELT. The effort was suspended during the week ending January 17th via this terse announcement.

GDELT Suspension

GDELT Suspension

There was an addendum to this which I didn’t include int he screen capture, but it mentioned Robin Kaler at the University of Illinois. I wrote her seeking additional information on the suspension and I received a response just moments ago.

GDELT Kaler

GDELT Kaler

“serious questions about the origins of the source texts used to code GDELT”

I believe this means that whomever created the CAMEO coding was either not credited appropriately, or there may be an issue with using it in a derivative product. I ran into this late last year – I was going to republish the Global Terrorism Database packaged for use with Sentinel Visualizer, but this was not allowed. I was free to publish a set of scripts to accomplish this task, the issue was that the entities that fund that effort wanted a count of total users, so any derivative work had to be post processing run by the user, rather than repackaging.

I hope what we are seeing here is some sort of pause to clean house and/or make things right with regards to whatever coding material was incorrectly used. The volume and quality of content was extremely promising and I hope the suspension is just some misunderstanding that can be quickly corrected. I kept the archive of the 1979 – 2012 data so I can continue working on something that will handle the live feed when it returns.

Muckety: Names, Nodes & Relationships

As a response to Visualizing Graph Databases With Linkurious I received a tip that I should look at Muckety. This system provides social/network visualization with data from a variety of important sources.

I have looked at The Militarist Galaxy as documented by RightWeb in the past so I choose one of the more noxious individuals, Frank Gaffney, as a starting point. I was pleased and surprised to see the Mother Jones articles on Groundswell were integrated.

Muckety Graph: Frank Gaffney

Muckety Graph: Frank Gaffney

Gaffney shares the Groundswell link with Allen West, a former Congressman and a war criminal who is equally willing to have journalists abused in this country. I dug just a little further than this and my eyes opened wide at the connections which were revealed.

Muckety: Frank Gaffney & Allen West

Muckety: Frank Gaffney & Alen West

A tool like Muckety doesn’t replace a desktop link analysis setup like Maltego, Gephi, or Sentinel Visualizer. What it does do in an easily accessible fashion is permit people who might otherwise never handle such technology to enter the name of an individual or organization that interests them, and immediately see important connections which would otherwise take hours of Googling, reading, and note taking.

Network analysis is to 2014 what social media was to 2009 – something that has specialists using it, but which will rapidly spread due to the powerful sense-making capabilities it offers when trying to understand complex interactions.

Visualizing Graph Databases With Linkurious

Yesterday a @kdnuggets tip led to SciCast: Gamified Technocracy and he hit another home run today with a pointer to Linkurious, a link analysis tool that works with the neo4j graph database. The beta signup leads to a live demo that contains much of the Internet Movie Database.

They suggested Clint Eastwood as an initial search term for the five minute demo, which walks you through the basics. Anyone acquainted with link analysis/data visualization tools like Maltego or Gephi will find the environment quite familiar. I continued playing after completing the demo, starting with Laura Dern, an actress David Lynch favors – a good choice as there are a lot of overlaps in casting between his various movies.

Linkurious Laura Dern

Linkurious Laura Dern


Linkurious Blue Velvet

Linkurious Blue Velvet


Linkurious David Lynch

Linkurious David Lynch

Just a few mouse clicks were required to hunt up actors, actresses, and movies. The system produces the links automatically. This is a web demo of what will be a desktop product that connects to neo4j. Pricing for a single user is similar to Maltego, enterprise is in line with the cost of a single seat for Sentinel Visualizer.

Linkurious Pricing

Linkurious Pricing

This is an exciting development for me. Maltego and Gephi are good for what they’re meant to do, but they have limits. Maltego entity types can be extended but the system starts to choke around the thousand node mark. Gephi can scale up to tens of thousands of nodes, but it is more general purpose, lacking the concept of node type. I had just picked up a free copy of the Graph Databases O’Reilly book earlier this week, which focuses on neo4j, and I am taking finding a visualization tool meant to work with it as a sign I am on the right track.

SciCast: Gamified Technocracy

A little over a year ago I surveyed the reelection of Barack Obama, the mix of incredulity and rage from the fringe right, and I knew that my work in this area was complete. The GOP faced the stark choice of trimming its sails for the demographic headwinds or being reduced to a regional party, but it took the shutdown of 2013 before they realized they were going to have to actively clean house rather than passively waiting for time to do the job.

I set out looking for something new to do at the start of 2013 and I knew it wasn’t going to be domestic. I spent a lot of time studying the social and professional network of Wikistrat, e-International Relations, and other foreign policy related collectives. The response when I approached was a pretty universal: “So, you are a college dropout and a hacker? Really?” I would not have predicted it at the beginning of the year, but the Terrorism Research & Analysis Consortium proved to be just the right size for my odd mix of skills and interests.

I was pleased with one anchor outlet like TRAC, but I see forces at work in the world, manifesting in social media, that indicate something is about to happen, but it’s hard to say exactly what. I happened to scan my Twitter timeline a bit ago, and one of those imminent somethings popped right out at me:

@kdnuggets hints about SciCast

@kdnuggets hints about SciCast

I took a look at the Twitter account for the effort and I was amazed that something so new had scored a mention from data mining guru @kdnuggets. I got in on the ground floor – follower #70.

@SciCasters Twitter

@SciCasters Twitter

What’s inside is as promising as a fledgling effort getting big name attention. Fourteen prediction areas, a clear sign of gamification in the form of a leaderboard, and it’s crowdsourced from the question creation through to the results.

SciCast Dashboard

SciCast Dashboard

When social media exploded onto the scene about five years ago the competing systems were clawing for market share. Twitter, Facebook, MySpace in decline except for music, Google+ trying to be ubiquitous and ending up intrusive and annoying, and a horde of also-rans that were mowed down by Facebook’s momentum and Twitter’s open format.

Fast forward five years, what social media platform am I using? Twitter, but it’s measured in minutes per day, as I passively observe leaders in various areas that interest me. I can go a month without logging into Facebook, but I have yet to make it a full quarter. LinkedIn is tolerable given the ability to lock up one’s contacts and mute chatterboxes.

Here are some trends I have noted:

One by one, email providers are requiring SMS verification before they’ll provide an account. The scammy, spammy nature of new social media accounts has much to do with this, as providers are seeing hordes of accounts registered and tended just long enough to make social media registrations work.

Someone gifted me a ScienceX profile and here I’m seeing one aspect of the future – people are willing to pay $15/year for high quality content and trolls aren’t willing to pay a $15 per ban tax.

The price of entry for SciCast is intangible but the bar is set very high – if you’re not serious about scientific method and quality control you’ll sink like a stone. They don’t explicitly state it but I’m sure they have sterner measures for anyone who is intentionally disruptive.

Stepping back even further, the very concept of the nation state seems to be in decline, but corporate power is far from assured. None of them run without humans on the inside, and Edward Snowden has made it painfully clear what happens when those people begin to question the nature of their work. And those corporations all have to participate in some fashion in this melange of social networking.

We evolved as a species on predator filled savannah, grouped together and task differentiated to maximize survival. Deprived of a secure place in the ranks of a union or on a stable corporate ladder, Randian theories glorifying hyper-individualism quickly fall away. The hunter-gatherer band has come back into fashion and networked humans are an acid bath for monolithic entities that draw the wrong sort of attention to themselves. SciCast is just the latest of a rainbow of threats to slow moving, internally politicized companies that have to meet shareholder expectations.

LinkedIn Reach & Engagement

I have been steadily rebuilding my LinkedIn presence since December of 2012, taking a weekly snapshot of the progress using the InMaps tool. A few days ago I published DDG-1000 Zumwalt: America’s New Destroyer, and it got picked up by someone outside my usual circle, scoring almost a thousand views.

LinkedIn Zumwalt Article Reach

LinkedIn Zumwalt Article Reach

I had never played with the article reach graph, but this motivated me to have a look at how my other updates there have done. I have a consistent audience of 20 to 40 viewers, so 5% to 10% of my 450 contacts. I think overall that’s a good number of people paying attention.

The other post that did well was Aegean Ethnographic Map. My core audience turned out, 28 of them, and 44 people who wouldn’t normally see my work did.

LinkedIn Aegean Ethnographic Map Update

LinkedIn Aegean Ethnographic Map Update

LinkedIn is subject to a curious dichotomy. There are people who try to do serious work and quality curation, a large number of low volume feeds. Unless you filter, they are utterly overwhelmed by ‘promoters’, spewing high volumes of low to negative value content. I have seen people in the “quantity over quality” intelligence related discussions posting articles from World Net Daily and other trash grade ‘news’ sites.

A small number of groups are properly policed and they provide quality content. Many group creators manage to grow an audience but they fail at group hygiene. There are many LinkedIn Groups, I only take email for five of the fifty to which I am subscribed, the rest are there because I permit fellow group members to contact me without being a connection.

Once upon a time I took an interest in readership volume. I think my best day ever several hundred thousand people read what I had written. The world has changed since then, Twitter has taken over, and in its turn Twitter is now overrun by largely automated systems in which a handful of skilled operators can drive the discussion in whatever direction they choose. Now it is better to have two or three dozen people who read and heed what is offered, rather than ten thousand times that many who can only spare a glance.

DDG-1000 Zumwalt: America’s New Destroyer

The last I knew of the DDG-1000 Zumwalt, our planned next generation of destroyers, the program had been canceled. I just noticed that this effort has been restored and the first ship of the planned three vessel series was floated in October of last year.

DDG-1000 Zumwalt

DDG-1000 Zumwalt

The United States currently has three types of surface combatants that do escort duty with Carrier Strike Groups and Expeditionary Strike Groups.

Oliver Hazard Perry Class FFG

Oliver Hazard Perry Class FFG

The Oliver Hazard Perry class frigates are a 4,100 ton displacement escort developed in the 1970s to replace World War II vintage destroyers. We built 71 and 13 remain in service. Their duties range from mine sweeping to anti-aircraft protection for other vessels.

Arleigh Burke Class DDG-51

Arleigh Burke Class DDG-51

The Arleigh Burke class guided missile destroyer are the fleet’s workhorse escort. There are three sub-types or ‘flights’ among the 62 vessels, which we began launching in 1989, all of which are still in service. Depending on the flight they displace between 8,300 tons and 9,800 tons. Armament includes 20mm Phalanx anti-missile system, 25mm Bushmaster for small surface threats and mines, and 5″ lightweight guns for long range bombardment.

Port Royale CG-73 Ticonderoga Class

Port Royale CG-73 Ticonderoga Class

Designed and Constructed between the Oliver Hazard Perry Frigtes and the Arleigh Burke Destroyers, the Ticonderoga class guided missile cruisers used the Spruance class destroyer hull and displaced just 9,600 tons. These were the first ship to be classified as a cruiser without having multiple dual or triple gun turrets and weapons of 6″ or larger caliber. The first five had twin arm missile carriers and were retired to home waters duty fairly quickly, while the 22 more capable ships with vertical launch systems remain in service today.

As our first 21st century surface combatant the DDG-1000 Zumwalt offers a number of interesting advances over its predecessors. Displacing 14,500 tons, they are nearly as large as our last gun cruiser, the Des Moines class, but radar return is reportedly similar to a fishing boat 1% of their size. The crew of 142 is half the size of the Arleigh Burke and Ticonderoga class ships.

The Zumwalt’s 80 vertical launch cells are less numerous than the Arleigh Burke (90 or 96) or the Ticonderoga (122) but they support missiles with 45% more ‘rocket motor mass flow rate’. I am unsure how to translate that into added thrust or payload, but the Zumwalt’s VLS is ready for longer range missiles we’ve not yet built. The reduced missile count doesn’t mean reduced ship capacity thanks to the presence of the Advanced Gun System.

The current 5″ lightweight gun employs a 20 round magazine and an autoloader to deliver a 70 pound conventional shell at distances of up to 23 miles. A World War II sailor would be impressed with the range and the ability to deliver 20 rounds in a minute, but they would find the work of the six man gun crew very familiar. The AGS would be utterly foreign to them, offering tremendous advances which include:

  • Automated handling of up to 600 rounds
  • Water cooled for continuous firing & improved barrel life
  • 155mm shell (50% more mass than current 5″ gun)
  • Smart ammunition

There isn’t much information available on the AGS via Wikipedia, but the literature on 155mm land artillery is instructive. Base bleed shells offer a 30% range increase over conventional rounds and glide capable shells push the range to 25 miles. The AGS round is apparently capable of reaching out almost 70 miles.

A conventional shell fired from 25 miles is unlikely to hit the correct city block. Field experience in Iraq indicate that there is a good probability that a smart round will hit the correct room in a given house – 90%+ will fall within four meters of their intended destination. The SMArt 155 delivers a pair of smart explosive formed penetrators that can easily take out a T-72 tank. This video shows a similar system in operation.

The DDG-1000 packs a pair of AGS and a trio of MQ-8 Fire Scout drones. These helicopter UAVs can linger for three hours at the extreme range of a gun system with pinpoint accuracy against fixed and mobile armored targets. Putting this in terms World War II history buffs will understand, one Zumwalt, one full, fresh Panzer division approaching the Normandy beachhead, one hour fire mission. 75% or higher loss rate for the armor. And they could do this from up to forty miles beyond the range of the largest shore defense guns in the area, 340mm naval guns salvaged from the French battleship Provence.

The U.S. Congress greatly extended the lives of the Iowa class battleships, returning them to service for Vietnam in the 1960s, Lebanon in the 1980s, and Desert Storm in the early 1990s, specifically citing their shore bombardment capability. The Zumwalt is the first shore bombardment capable ship we’ve launched since the U.S.S. Newport News, sixty five years ago.

I would prefer from a policy perspective that we focus on proactive diplomatic problem solving rather than reactive interventions, but there are some problems which require direct, forceful action. On those occasions when we must deploy an Expeditionary Strike Group, the Zumwalt will be a significant upgrade over our current destroyers and cruisers.

Methamphetamines Funding The Syrian Insurgency

Six months ago in Funding The Syrian Insurgency I noted the conflict in northeast Syria regarding control of the oil fields. Those wishing to understand the importance of the connection between insurgency and the illicit networks that fund them should look at Paul Collier & Anke Hoeffler’s Greed & Grievance in Civil Wars (pdf).

Today I noticed Insight: War turns Syria into major amphetamines producer, consumer, which reveals an interesting set of interlocking issues. The trade itself is apparently producing hundreds of millions of dollars in revenue, providing income for both the government and rebel forces, as well as fueling the duration and intensity of street battles.

I have more digging to do before I can make any sort of sensible characterization of what is happening. This post is going to be an inventory of what I believe to be relevant, and it will likely be rather disjoint.

Five months ago I wrote Yemen’s Food & Water Crisis. Residents of the Horn of Africa and the Saudi Peninsula use khat, a mild stimulant that is consumed by chewing the fresh green leaves of the plant. Khat is a thirsty but profitable crop, being grown at the expense of food production in Yemen.

Khat’s active ingredient, cathionone, breaks down within about 48 hours after harvest, so the leaves must be chewed when fresh. The methylated form of this naturally occurring compound has similar effects to methamphetamine and it is a small but dangerous component of the overall stimulant abuse problem in the rural U.S. The Horn of Africa and Arabian Peninsula have a cultural norm of consuming a stimulant somewhat stronger than coffee, an entry point for more dangerous substances, such as methamphetamine dressed up to look like the milder Captagon, or phenethylline, a popular drug in the region.

Bulk methamphetamine production requires the availability of ammonia, the world’s most common industrial chemical. We make around 140 million tons a year globally, 30% with hydrogen from natural gas, mostly in the west, and 70% using hydrogen from coal. This is a value added product produced using stranded gas pools, notably in Trinidad in this hemisphere, formerly in Punta Arenas, and U.S. ammonia plants have been shut down, packed up, and reassembled in natural gas rich Qatar. Ammonia is easier to transport than compressed or liquified natural gas.

Syria has two large nitrogen production facilities located at the same geographic location, near Homs. Although the link providing the coordinates is about sulphuric acid production, other sources indicate this is also an ammonia production facility. Ammonia is a precursor for ammonium sulfate and a plant that first extracted sulfur from petroleum coke, then gasified it to make hydrogen would be a normal setup near an oil refinery. The nearby water source is also consistent – ammonia plants produce large volumes of low grade heat that is discarded via water fed cooling towers.

General Fertilizer Company Plant, Homs, Syria

General Fertilizer Company Plant, Homs, Syria


General Fertilizer Company, Homs, Syria

General Fertilizer Company, Homs, Syria

The United States banned ammonium nitrate sales after the 1995 Oklahoma City bombing. This chemical is still our most common fertilizer, but it’s delivered as a solution known as UAN, urea/ammonium nitrate, which is not usable as a component of explosives. The other delivery methods are as anhydrous ammonia, a cryogenic liquid and as urea alone, a water activated solid. Lesser amounts of ammonium phosphate and ammonium sulfate are used when soils need these elements as well as nitrogen.

The link between ammonium nitrate, agriculture, and insurgency produced explosives is a fairly intractable problem. Smallholders in developing countries don’t have the infrastructure to handle liquid UAN, let alone a cryogenic gas. They need nitrogen fertilizer in a bagged form. Urea is a solid, but it has to be applied just before or during rain in order for it to work. This doesn’t work in relatively arid places, so ammonium nitrate is still the solid fertilizer of choice. Producers have attempted to address this issue by making calcium ammonium nitrate, but recovering ammonium nitrate from it is a fairly simple chemical reaction.

That is what I think I know, here are some “known unknowns” that would help sharpen the overall analysis if they can be resolved.

  • Is Homs indeed the site of Syria’s in-country nitrogen production?
  • Who controls the plant? The Homs area? Who is in a position to divert ammonia?
  • Based on the Reuters article, global Captagon consumption is 21 tons. A tiny ammonia plant will produce a hundred tons a day.
  • How much, if any, of the plant’s output is ammonium nitrate?

The last point is important. There are many reports of Syrian regime helicopters dropping ‘barrel bombs’. These are 55 gallon drums, old water heaters, or lengths of iron pipe. The first video shows a string of devices employing parachutes to retard their fall. This is sometimes employed to permit a low flying aircraft to escape a large blast, but I believe this might be a strategy to reduce the failure rate of these hastily constructed IEDs, which use improvised impact fuses.

The second video shows a large blast that begins with a rolling cloud of flame and smoke. Hollywood dramatizes action sequences by using incendiary charges in place of actual high explosives, which often look like this, but high explosives don’t cause effects like this unless they hit something that has a liquid fuel supply. If the regime is making ammonium nitrate/fuel oil bombs and they’re adjusting the mixture for incendiary as well as blast effects, the use of incendiaries against civilian populations is a war crime.


Syria’s civil war has been understood in the west as conflict between the following:

  • Alawite/Shiite versus Sunni
  • Assad regime versus the Islamic State of Iraq and Syria
  • Former Soviet client state versus western friendly nations

While those three are all somewhat true, the missing component is the haves versus the have nots. People who are treated unfairly protest, if they’re starved as well they will engage in open revolt against the administration that is failing to meet their minimum needs.

Syrian Wheat Outlook on 12/8/2012 was my first inspection of agriculture in the country. Seven months later the New York Times provided information that led to Syrian Wheat Becomes Strategic. Food security in the country is a complex issue, but if you can only watch a single metric, wheat availability, price, and quality are a good proxy.

Attempting to stop thousands of foreign fighters by direct attack is the most destructive strategy of all for Syria. Stopping the flow of arms and explosives into the country, as well as interdicting the repurposing of domestic fertilizer into explosives attacks the problem at a lower level.

If the region’s entire Captagon habit were supplanted by methamphetamine it would only require one tenth of the daily production from a tiny ammonia plant to fill the requirement. If the diversion of ammonia is happening and it can be pinched off at the source this defunds a portion of the conflict.

The most likely entity that could protect an ammonia production source and divert a portion of it to drug production is … the Assad regime. If it were a rebel facility and pair of helicopters dropping a string of those parachute bombs would easily disable it.

It is functionally impossible to separate the production of food and the making of IEDs in arid regions where production depends on smallholders using bagged fertilizer. Ponder that concept while I go off and dig deeper into Syrian agriculture.