New API Allows Developers to Find Millions of User Comments Across the Web

Diffbot is an artificial intelligence startup and creators of visual learning robot technology that lets developers analyze, extract, and enhance Web content. Diffbot offers a robot that examines the Web using computer vision and natural language processing providing developers with tools to find, extract and understand the objects from any Web page for use in their applications. 

The company has released a new “Discussions API” to provide developers a tool to unlock the millions of comments made every day online in these formerly hidden corners of the Web. The new APIs allow developers to treat forums, comment threads and review collections as virtual databases, accessing their data on-the-fly. 

This user-generated content makes up a part of what is known as the “deep web,” estimated to be 400 times larger than the “surface Web” that is indexed and accessible through traditional search engines. The Discussion API currently supports Facebook Comments, Disqus, Livefyre, Wordpress, Blogger, Intense Debate (owned by Automattic), Kinja, Hacker News, Reddit, and more.

This new functionality visually analyzes web pages and instantly parses complete comment data, author information, topic analysis, and more into discrete objects. That structured data, now accessible via automation, provides on-demand access allowing developers and companies to build applications that can:

- Monitor brands, products or other keywords to gauge user reaction, gather feedback or monitor sentiment in the locations where users are providing feedback.

- Make forum and other user-created content mobile friendly for easier consumption on phones, tablets or other devices.

- Analyze user-created posts to identify trends or sentiment within user communities and among power users.

- Identify links and other content shared within discussion threads or comment sections to improve product recommendations, insert affiliate linking, and perform other link analysis.

- Completely process and extract an entire site’s worth of content, whether for migration or other analysis, when paired with Crawlbot, Diffbot’s intelligent spider.

The new API augments Diffbot’s efforts to structure the data of the entire Web, including its APIs for automatic article, product, image and video extraction; its Analyze API, which immediately determines the “page type” of any unknown link; and its Crawlbot crawling platform for entire-site extraction.