Data lifting and why it has to be made easy

At the end of 2017, there will be 8.4 billion connected things in use worldwide up 31 percent from 2016, and this figure is expected to reach 20.4 billion by 2020.

When Internet of Things (IoT) as an industry took off in India, it spawned a host of startups selling edge devices that could gather and crunch data from corporate customers.

These startups ran into one fundamental problem, which was data lifting. The data was so voluminous that these startups took so much time to organise them that they ran out of money to keep the companies afloat.

In the end, their services were just organising data for customers with very little insights.

At the Amazon AI conclave held recently, Amazon Web Services (AWS) launched a slew of services that will make data lifting and organising easy.

“Developers can now focus on building solutions through AI rather than just spending hours organising data”, says Anima Anandkumar Principal Scientist at Amazon AI, a division at Amazon.

The evolution of the industry along with the software industry has been through the cloud, which gave rise to the big data opportunity and from there it went on to machine learning and Artificial Intelligence (AI).

Machine Learning (ML) is available in so many aspects of Amazon. If we buy a book there is ML in recommendations, and Amazon is working on algorithms that automates inventory management. It is also working on drones to deliver efficiently and has released Lex as a voice and customer interactions service through Alexa.

Then there is Amazon Go, which is the future of the retail store where everything from payment to item picking is automated. Amazon has been able to achieve this because of its ability to understand enterprise problems. It has figured out that today there is a shortage of talent in data modelling and crunching.

“Today companies don't have ML practitioners and that's why we have created platforms to stop this laborious data organising and deploying processes,” says Anima.

With AWS the platform allows a developer to set a few rules and it factors all the data by itself. On top of that developers can choose an algorithm to build a data model. Once this data is organised and a leaning model is built on top of it, engineers and data scientists can create models for real-time decision making on deciphered data patterns. Think about the numbers of engineers that will be freed to build AI data model.

“Remember the past when there was so much money wasted on repetitive tasks, where an enterprise managed so many programmers to build and operate data. There was a lot of costs involved in this,” says Anima.

This is why it makes sense to work on readymade frameworks and interfaces.

So what is the solution for AI? This is where AWS Sagemaker is used. In this framework, data is being collected in a template format. After which a developer can use an algorithm. You can optimise the parameters and train the algorithms to learn from data patterns. Then the developer can deploy the AI model into production faster. It works best when you are deploying thousands of IoT devices in client locations. Insights can be given faster to the client and ensure cost savings.

Gartner, Inc. forecasts that 8.4 billion connected things will be in use worldwide in 2017, up 31 percent from 2016, and will reach 20.4 billion by 2020. Total spending on endpoints and services will reach almost $2 trillion in 2017.

Regionally, Greater China, North America and Western Europe are driving the use of connected things and the three regions together will represent 67 percent of the overall Internet of Things (IoT) installed base in 2017.

So why build on AWS?

It offers all layers of service from the application layer, to the platform and later, the infrastructure.

It has launched over 20 AI services and these tools are all meant to reduce the drudgery of sorting data fast. According to Gartner, it has a 44 percent market share in the cloud market and they are far ahead of Microsoft and Google. They have clients like Goldman Sachs, Digital Globe and NASA.

“Machine Learning along with data science offers scale and provides new capabilities and insights. We, at Expedia, are helping the customer gather insights from millions of price changes in the travel industry and offering him the best choices,” says Matt Fryer, head of data science at Expedia.com

In the 60s, travel agents had power; then the internet offered price discovery, and now you can navigate the complexities of the best price with AI and Machine Learning. The core is the cool basics like compute, elasticity and cost economics, which have all fallen thanks to the cloud. It transforms how we tackle problems. “We have a culture of experimentation and AWS helps us with its new AI and ML tools,” says Fryer.

No wonder the AI Conclave was packed and filled with millennials that want to use Amazon's tools to ensure easy data sorting and providing creative insights.