Amazon Alexa can easily play a requested song or tell you the weather, but no one expects her to analyze road and vehicle conditions when providing travel guidance. The reason we expect so little from apps like Alexa (or Siri or Google Home) is because these virtual assistants still use traditional methods of machine learning to respond to simple queries, rather than accounting for contextual information like human brains do.
In the enterprise, this type of data analysis can’t support business decisions because the demands of dynamic companies for continuous intelligence derived from streaming data sources is too great. Swim, a company building contextual analytics platforms for streaming data, is helping developers build applications specifically for this type of environment, where analytics can be done in real-time, at the source.
We spoke with Simon Crosby, CTO of Swim, about the things to keep in mind when building the next generation of enterprise apps. Simon discusses continuous intelligence and how app developers can best take advantage of streams of data from endpoint devices and sensors in real-time.
Crosby: Whereas cloud-native stacks build on a powerful triumvirate: Stateless APIs, microservices, and stateful databases, continuous intelligence needs different abstractions:
Crosby: There are hundreds of powerful databases to choose from, but there’s a limit to how much “intelligence” one can expect of a database engine. Databases don’t run applications and can’t make sense of data. And they are typically a slow network hop away from application logic. The trend is toward in-memory stores, grids and caches. But -
Crosby: Apache Kafka is a popular event streaming platform that performs a key role: It enables any number of sources to publish events about topics to a broker, and any number of applications to subscribe to topics. But brokers don’t execute applications - they act as a buffer. Instead, an event stream processor analyzes the stream: Ultimately the reasoning about the meaning of an event or set of events requires a stateful system model, and insights that rely on logical or mathematical relationships between data sources and their states, are not the domain of the broker.
Crosby: Best known for delivering widget views and KPIs, streaming analytics takes an approach based on regular computation of insights and delivery of the results to UIs used to understand the system at a “summary” level. A large number of tools exist in this category, both general-purpose and vertically integrated.
Managers of large systems - whether containers in the cloud or fleets of trucks, need detailed performance metrics specific to their use-cases. They are operations- and outcome-focused - they want to “zoom in” on issues to see problems - often in real-time. The meaning of data is critical so in streaming analytics the meaning of an event is always well defined.
Managers aren’t the only users though: Modern SaaS applications need to deliver contextually useful responses to end-users. Timely, granular, personalized and local responses are crucial, even for applications at massive scale. This kind of capability is beyond the scope of streaming analytics applications.
Streaming analytics applications use a defined schema or model, combined with business logic to translate event data into state changes in the system model. The meaning of state changes triggered by events is the “business logic” part of an application and has a specific interpretation given the use case. For this reason streaming analytics applications are often domain specific (for example in application performance management), or designed for a business specific purpose.
Crosby: Applications that demand continuous intelligence at scale are characterized by a need to understand streaming data in context, and respond — in real-time, and at scale. They fuse streaming and traditional data, analyzing, learning, and predicting on-the-fly in response to streaming data from many sources. They
The flow of continuous intelligence and automatic responses has to be computed and responses delivered concurrently. Tens or hundreds of millions of complex evaluations may need to be executed at the same time — for each “thing” and all of its current relationships.
Finally, the results of analysis, learning and prediction must be visually available at all levels of detail. A KPI that shows aggregate information, such as all traffic lights in a city, should update in real-time to reflect what’s happening on the ground. Real-time visualization is a critical aspect of continuous intelligence.
Simon Crosby is CTO at Swim. Swim offers the first open core, enterprise-grade platform for continuous intelligence at scale, providing businesses with complete situational awareness and operational decision support at every moment. Simon co-founded Bromium (now HP SureClick) in 2010 and currently serves as a strategic advisor. Previously, he was the CTO of the Data Center and Cloud Division at Citrix Systems; founder, CTO, and vice president of strategy and corporate development at XenSource; and a principal engineer at Intel, as well as a faculty member at Cambridge University, where he led the research on network performance and control and multimedia operating systems.
Address:
3003 East Chestnut Expy
STE# 575
Springfield, Mo 65802
Phone: 1-844-277-3386
Fax:417-429-2935
E-Mail: contact@appdevelopermagazine.com