A Hewlett Packard
success story

Hewlett Packard Enterprise transforms customer experience with near real-time insights

The need

HPE’s vision for helping customers is to slash the time it takes to turn ideas into value, which is transforming industries, markets, and lives. The company’s InfoSight product aims to deliver on this vision by providing customers with insight into their infrastructure in seconds and minutes, rather than hours or days.

InfoSight works by receiving infrastructure information from a network of over 20 billion sensors deployed at data centers all around the globe, which send trillions of metrics to InfoSight every day.

“We’re watching what’s going on in the environment and putting the data through expert systems using various AI and machine learning techniques,” says Jeff Dutton, Data Platform Architect for HPE InfoSight. “The goal is to tell the customer what to do to prevent problems and optimize their experience.”

The challenge

Increasing demand from customers and internal stakeholders at HPE to deliver insight in real time necessitated a major overhaul to the InfoSight architecture, as Dutton explains:

“Moving to a near real-time user experience requires an infrastructure that self-heals, scales massively and is always available to process streaming workloads, no matter what. This is fast becoming table stakes for every business.”

The speed by which Dutton and the team could deliver insights to customers was gated by the time it took to run batch processes against very large, extremely complex data sets. To deliver value faster, InfoSight needed to evolve beyond its classic, batch mode, big data architecture.

I have been working with Akka since 2014 and knew they could deliver on the elasticity and resilience we required.

Jeff Dutton
Data Platform Architect. HPE InfoSight

The solution

As an initial experiment, the team migrated from an enterprise database to Apache Spark to see if faster batch processing was the answer. However, they quickly realized they would need to transition to work with the data in real time to realize their business vision. This meant introducing stream processing.

“Batch analytics is something nearly every developer and architect understands fundamentally, regardless of their tech stack,” notes Dutton. “Batch continues to play an important role in the InfoSight architecture; it’s leveraging several petabytes of historical data. The source for near real-time insight, however, becomes the stream. And it’s the streaming part of the journey that has been the most challenging.”

Dutton turned to Akka for the elasticity and resilient self-healing required to deliver big data at speed.

“I have been working with Akka since 2014 and knew they could deliver on the elasticity and resilience we required,” shares Dutton. “When they packaged their Akka platform stack with Apache Kafka, Apache Spark, and a multitude of other technologies into a de facto fast data distribution, it made good business sense to use their platform for the next generation of InfoSight.”

The results

Akka provides HPE with microservices frameworks for processing continuous application logic; multiple streaming engines for handling tradeoffs between data latency, volume, transformation, and integration; machine learning and deep learning tools for applying algorithms from the data science team; and intelligent management and monitoring tools for reducing the risk of running this always-on system in production.

Data services in a dynamic, streaming application must run forever and be able to scale horizontally across multiple nodes. This is a fundamental advantage of Akka’s concurrency model, which is built around independent, self-contained processes that communicate asynchronously via messages. This means that they can be distributed and run across many machines.

In addition to scaling elastically, Akka enables the HPE InfoSight streaming applications to recover from failures quickly. Failure is treated a first class concept: failures are normal, anticipated, and handled within the application, which enables self-healing.

Dutton’s approach is fast becoming a reference for other HPE business units looking to deliver similar real-time customer value. The solution that he and his team have built with Akka for HPE InfoSight is making such sound business sense that HPE Storage and HPE Pointnext are now teaming to bring the streaming solution to HPE’s top customers.

MORE SUCCESS STORIES

Stay Responsive
to Change.

GET STARTED
CONTACT US