By Scott Rose, Vice President of Services, Think Big
What does Think Big have to say about the Internet of Things (IoT)? As it turns out, quite a bit. Since the company’s inception five years ago, we have worked on roughly 100 projects—with approximately 20 centered on IoT. By far the largest projects we’ve completed, they typically begin with a big data strategy, move through several engineering releases, and then transition to longer-term support. We have therefore seen first-hand the business impact IoT solutions deliver to organizations from virtualizing and analyzing their physical world.
Think Big doesn’t engineer end-to-end IoT solutions. We build out the last mile—starting a project once sensors are in place and generating data. In most cases this is existing data that hasn’t been used for larger scale analytics, such as machine logs off the manufacturing plant or device logs from product in the field. Data historically used by engineers for R&D or to fix technical problems. Forward thinking companies are now using this data to innovate their businesses, for example to re-engineer their manufacturing and support processes or to invent new services.
Simply put, we develop solutions that collect device data, put it into a central data store, and make it available for analysis. In doing this Think Big has first-hand experience with the technical and business problems organizations face launching and scaling these programs, and continue to work with a number of customers to solve these challenges.
IoT Implementation Challenges
So what sort of challenges do we typically see with IoT implementations? I surveyed our field project managers and asked them to tell me their best war stories. It was clear from our discussions that the technology innovation is real, with performance and economics that easily justified moving forward with IoT investments. I heard about common engineering challenges such as data quality, and our clients’ struggles in building the software engineering and data science skills big data platforms require. I also heard about common business challenges, specifically the change management required to reliably operationalize the insights businesses get from these new systems.
Many of our customers start by trying to implement an IoT solution on their own with existing technologies, leveraging skills they already have. Inevitably they hit a wall. As an example, we recently worked with an insurance company on a “real-time” rating program. I put “real-time” in quotes because performance constraints in the platform they used led to a four-month delay processing telematics data.
This defeated the point of the entire program. For example one of their customers could drive recklessly for months before their insurance rates changed. For the insurer, the main issue was capacity on the current system and the cost to add more storage and processing. The solution? Think Big helped them re-platform the system to Hadoop to get acceptable performance—processing the data in less than a day at a much lower cost per terabyte.
For IoT solutions the technical challenges are not only due to processing orders of magnitude more data in the core data management and analytics platforms. At the edge, IoT devices often have limited memory and intermittent network connections, so a lot of data goes missing. Our insurance client’s telematics system often lost data that led to inaccurate trip profiles. For example, a drive to the grocery store could go on for days and thousands of miles. The solution needed to have the resiliency to flag and recover from these data quality issues where possible, and adjust analytics processing where it was not.
My interviews with our teams building IoT systems revealed that business issues are more critical to the longer-term adoption and health of these programs than technology issues. This is, of course, not to diminish technical challenges; however, I believe they are more or less solvable today with custom development or soon with a couple turns of Moore’s Law. The business issues are much more daunting, and I will address those in my next post.