- Joined
- Jun 27, 2006
- Location
- Chicago, IL
IoT is all the rage, but how do you connect those "things" TO the internet, to the cloud, to Azure? Luckily Visual C++ MVP Alon Fliess is here to help with a great blog post.
Introduction to the Internet of Things – From the Device to Microsoft Azure Cloud
Follow @CH9
Follow @coding4fun
Follow @gduncan411
Continue reading...
Introduction to the Internet of Things – From the Device to Microsoft Azure Cloud
Introduction to the Internet of Things – From the Device to Microsoft Azure Cloud
Technology advances in “Buzzwords” steps. At the beginning, there is the basic technology. It slowly evolves, and then after a few years, sometimes even many years, everything becomes connected and the world, not the early bird world, but everybody is ready to embrace that technology. This is where the big buzz begins and everybody predicts that in five to ten years the technology will generate incremental revenue exceeding hundreds of billions. With this buzz, all major companies invest in the technology and we begin seeing TV news reports and economy magazine articles about the technology, telling that the everyday life of every human being on the planet is going to change because of that technology!
Of course, I am a bit cynical, but this is exactly what is happening now with regards to the IoT – the Internet of things. The basic technology is already here for almost a decade so far. Amazon Web Services (ASW) started in 2006. Microsoft Azure is 5 years old. Devices such as those based on the Amtel AVR controller are more than 20 years old, and the affordable Arduino family of devices used by IoT hobbyists are 10 years old. It is not (just) the technology that makes IoT what it is, but the concepts, the perception, the commitment and the challenges that the entire industry is dealing with nowadays. IoT is about the machine-to-machine (M2M) communication at scale. Vast numbers of devices using different hardware and software technologies are connected between them and to the cloud. The cloud provides many services, which can handle huge streams of data, analyze and can extract vital information about the current state of the system and can even predict future state.
So what exactly is IoT?
In one simple form, IoT is about a device that can monitor a physical character of the environment and transfer this data over the Internet to a cloud service. In a more complex form, IoT is the combination of many smart devices that can “feel” the environment, read the data, and transfer this information to a collector service in the cloud. This service has to deal with large amounts of devices and huge streams of data. Such services take in information and can extract a vital information from a live stream, or run algorithms such as those based on big-data map-reduce patterns. Services can act on historical and new data, or can provide future insight about the collected data. Services like Azure Machine Learning can use the collected data to predict future behaviors. The cloud can send commands to devices. Take for an example a system that starts the water sprinklers according to an algorithm that reads information from a group of soil moisture level sensors. Based on information that it gets from a forecast service that predicts that no rain is coming, it decides to send a command to an actuator that starts the sprinklers.
Sometimes the end device has the capabilities to communicate directly with the cloud service, and sometimes the device is cheap, weak, and has to conserve power, or has no encryption capabilities. In the latter case, a group of such weak devices is connected to a local gateway – a software that runs on a stronger local device that serves as a mediator between the local device sub-network and the cloud.
To illustrate, let us take a simple personal and fun project – which I have only recently turned into an IoT projectJ). With this project, I can control my home electrical devices such as lights, shutters, hot water boiler, and the air condition. I can use a web browser, Windows Phone or Windows 8.1 Modern application to send a command or read the current state of a device. I have other services that turn on the garden lights on sunset, or raise the shutters automatically every morning except on weekend days. I even have a service that sets the boiler according to the forecast.
...
An IoT System
Every device that has a way to communicate can take part in an IoT system. A modern IoT system is built from at least one end device and one service that runs over the internet, however most large IoT systems include one or more of the following elements:
1. Many different devices with sensors and actuators
2. Local gateways
3. A collection of cloud services that enables:
a. Registration of devices
b. Management of devices
c. Different communication protocols that provides reliability and security
d. The ability to collect a vast amount of data at a very high input rate
e. The ability to analyze the stream of information in close to real-time manner
f. The ability to analyze the current and historical collected information
g. The ability to show the resulting conclusion and the collected data
In the rest of the article, we will see the two sides of an IoT system. We will start with the device and we will continue with the cloud services focusing on the rich possibilities that Microsoft Azure provides such a system.
The Devices:
...
Raspberry Pi 2 – Will run Windows 10
There are other Raspberry Pi compatible but more powerful devices such as the Banana-Pi and the Orange-Pi. The Orange Pi has a built-in Gigabit Ethernet and Wi-Fi, which make it a good IoT (gateway) device.
An Orange Pi
Intel supplies both hardware and software for IoT. The Intel Galileo 2 board is a relatively robust machine that runs Linux or the new Windows IoT (still in alpha version). The Galileo is compatible with Arduino shield and software and you can use Visual Studio C++ and sub-set of the Win 32 API. With these C++ and Win 32 capabilities, you can use the C++ REST SDK and Azure C++ Storage API with the Galileo to communicate with Microsoft cloud.
...
Smart Refrigerator Scenario
We would like to have a set of devices and a backend system that transforms our refrigerator to an IoT system. The system can tell us that we are running out of milk, it can tell our retail store that they need to deliver a new milk bottle, and with the power of the cloud it can tell the retail chain headquarter the trends of using their goods and predict the near future supply needs. This knowledge provides real business value. There are many different ways to make a smart-refrigerator – for example, each product can come in a smart package that knows the product details such as the expiration date, the price, the nutrition and the current amount that is in the box. For our humble example, we are going to use a device that can weigh the milk bottle; we will convert the result to the current amount that is left in the box. We can set a collection of such devices in the refrigerator for different products.
A prototype setup looks like this:
...
The Gateway and the Cloud:
Up until now, we have seen all about the device, but IoT is mainly about connecting devices to the cloud and handling the stream of data.
Microsoft Azure provides the required framework to build a large-scale IoT backend:
...
Azure Event Hub
Azure Event Hub is a highly scalable service that enables ingesting and processing millions of events per second. Queues and topics are great for building reliable and scalable enterprise solutions but event hubs are particularly suitable to the IoT pattern of scale. In any scenario that involves scalability – you must tackle the potential bottlenecks of the system. Queues, topics and event hubs handle message throughput differently. Queues synchronize producers and consumers – for example, queues provide a mechanism that hides a message once a consumer starts to handle it, a mechanism to delete it when the consumer finishes handling it, or the consumer can return the message to the queue if it could not process it to completion. Topics are similar, with the exception that multiple consumers can process a message. To enable improved scalability, low latency and performance, event hubs have a different structure and approach. Event hubs have many data structures similar to queues called partitions. When you create an event hub, you provide the number of partitions – currently up to 32, but you can request additional partitions if this is insufficient for your solution. Each message sent to the event hub can include a target partition but you can also use a key such as a device id and allow the event hub infrastructure to co-located messages from the same device id to the same destination partition. In case of the latter – the device id goes through a hash function that chooses the partition for its stream of data. This enables scalability and message order from a device source. To further increase the performance, event hubs do not delete events immediately on consumption. In fact they don’t track the consumer state at all. Instead, event hubs use the notion of consumer groups. There is at least one group – the default consumer group, however in many cases you would create more than one consumer group to scale out or add alternate processing behaviors. Each consumer group tracks the location of the reading head – to track the location of the next event. There is one tracking cursor per partition. Event hubs delete messages based on the time-to-live setting on the message. You can supply a time-to-live for a message from one to seven days. Since messages are not automatically removed upon consumption, you can create multiple consumer groups to read and process past messages – a handy feature that lets you analyze a range of messages.
...
Getting Data from Azure – Microsoft Power BI Tool
You can watch the complete demo video.
Summary
Internet of Things is much more than hype. You can leverage the technology to get vital information to drive your business. Companies are investing in IoT in a variety of ways. Some are building better IoT frameworks, protocols, services and systems. Others are looking at providing IoT services or looking at how they can use IoT systems to improve their current solution. A Security related companies will provide security for IoT devices and services, billing companies may evolve to provide solutions for using IoT devices and services, and retail companies – they will know when to deliver a new bottle of milk!
... [Click through for the entire post, the code and more]
Technology advances in “Buzzwords” steps. At the beginning, there is the basic technology. It slowly evolves, and then after a few years, sometimes even many years, everything becomes connected and the world, not the early bird world, but everybody is ready to embrace that technology. This is where the big buzz begins and everybody predicts that in five to ten years the technology will generate incremental revenue exceeding hundreds of billions. With this buzz, all major companies invest in the technology and we begin seeing TV news reports and economy magazine articles about the technology, telling that the everyday life of every human being on the planet is going to change because of that technology!
Of course, I am a bit cynical, but this is exactly what is happening now with regards to the IoT – the Internet of things. The basic technology is already here for almost a decade so far. Amazon Web Services (ASW) started in 2006. Microsoft Azure is 5 years old. Devices such as those based on the Amtel AVR controller are more than 20 years old, and the affordable Arduino family of devices used by IoT hobbyists are 10 years old. It is not (just) the technology that makes IoT what it is, but the concepts, the perception, the commitment and the challenges that the entire industry is dealing with nowadays. IoT is about the machine-to-machine (M2M) communication at scale. Vast numbers of devices using different hardware and software technologies are connected between them and to the cloud. The cloud provides many services, which can handle huge streams of data, analyze and can extract vital information about the current state of the system and can even predict future state.
So what exactly is IoT?
In one simple form, IoT is about a device that can monitor a physical character of the environment and transfer this data over the Internet to a cloud service. In a more complex form, IoT is the combination of many smart devices that can “feel” the environment, read the data, and transfer this information to a collector service in the cloud. This service has to deal with large amounts of devices and huge streams of data. Such services take in information and can extract a vital information from a live stream, or run algorithms such as those based on big-data map-reduce patterns. Services can act on historical and new data, or can provide future insight about the collected data. Services like Azure Machine Learning can use the collected data to predict future behaviors. The cloud can send commands to devices. Take for an example a system that starts the water sprinklers according to an algorithm that reads information from a group of soil moisture level sensors. Based on information that it gets from a forecast service that predicts that no rain is coming, it decides to send a command to an actuator that starts the sprinklers.
Sometimes the end device has the capabilities to communicate directly with the cloud service, and sometimes the device is cheap, weak, and has to conserve power, or has no encryption capabilities. In the latter case, a group of such weak devices is connected to a local gateway – a software that runs on a stronger local device that serves as a mediator between the local device sub-network and the cloud.
To illustrate, let us take a simple personal and fun project – which I have only recently turned into an IoT projectJ). With this project, I can control my home electrical devices such as lights, shutters, hot water boiler, and the air condition. I can use a web browser, Windows Phone or Windows 8.1 Modern application to send a command or read the current state of a device. I have other services that turn on the garden lights on sunset, or raise the shutters automatically every morning except on weekend days. I even have a service that sets the boiler according to the forecast.
...
An IoT System
Every device that has a way to communicate can take part in an IoT system. A modern IoT system is built from at least one end device and one service that runs over the internet, however most large IoT systems include one or more of the following elements:
1. Many different devices with sensors and actuators
2. Local gateways
3. A collection of cloud services that enables:
a. Registration of devices
b. Management of devices
c. Different communication protocols that provides reliability and security
d. The ability to collect a vast amount of data at a very high input rate
e. The ability to analyze the stream of information in close to real-time manner
f. The ability to analyze the current and historical collected information
g. The ability to show the resulting conclusion and the collected data
In the rest of the article, we will see the two sides of an IoT system. We will start with the device and we will continue with the cloud services focusing on the rich possibilities that Microsoft Azure provides such a system.
The Devices:
...
Raspberry Pi 2 – Will run Windows 10
There are other Raspberry Pi compatible but more powerful devices such as the Banana-Pi and the Orange-Pi. The Orange Pi has a built-in Gigabit Ethernet and Wi-Fi, which make it a good IoT (gateway) device.
An Orange Pi
Intel supplies both hardware and software for IoT. The Intel Galileo 2 board is a relatively robust machine that runs Linux or the new Windows IoT (still in alpha version). The Galileo is compatible with Arduino shield and software and you can use Visual Studio C++ and sub-set of the Win 32 API. With these C++ and Win 32 capabilities, you can use the C++ REST SDK and Azure C++ Storage API with the Galileo to communicate with Microsoft cloud.
...
Smart Refrigerator Scenario
We would like to have a set of devices and a backend system that transforms our refrigerator to an IoT system. The system can tell us that we are running out of milk, it can tell our retail store that they need to deliver a new milk bottle, and with the power of the cloud it can tell the retail chain headquarter the trends of using their goods and predict the near future supply needs. This knowledge provides real business value. There are many different ways to make a smart-refrigerator – for example, each product can come in a smart package that knows the product details such as the expiration date, the price, the nutrition and the current amount that is in the box. For our humble example, we are going to use a device that can weigh the milk bottle; we will convert the result to the current amount that is left in the box. We can set a collection of such devices in the refrigerator for different products.
A prototype setup looks like this:
...
The Gateway and the Cloud:
Up until now, we have seen all about the device, but IoT is mainly about connecting devices to the cloud and handling the stream of data.
Microsoft Azure provides the required framework to build a large-scale IoT backend:
...
Azure Event Hub
Azure Event Hub is a highly scalable service that enables ingesting and processing millions of events per second. Queues and topics are great for building reliable and scalable enterprise solutions but event hubs are particularly suitable to the IoT pattern of scale. In any scenario that involves scalability – you must tackle the potential bottlenecks of the system. Queues, topics and event hubs handle message throughput differently. Queues synchronize producers and consumers – for example, queues provide a mechanism that hides a message once a consumer starts to handle it, a mechanism to delete it when the consumer finishes handling it, or the consumer can return the message to the queue if it could not process it to completion. Topics are similar, with the exception that multiple consumers can process a message. To enable improved scalability, low latency and performance, event hubs have a different structure and approach. Event hubs have many data structures similar to queues called partitions. When you create an event hub, you provide the number of partitions – currently up to 32, but you can request additional partitions if this is insufficient for your solution. Each message sent to the event hub can include a target partition but you can also use a key such as a device id and allow the event hub infrastructure to co-located messages from the same device id to the same destination partition. In case of the latter – the device id goes through a hash function that chooses the partition for its stream of data. This enables scalability and message order from a device source. To further increase the performance, event hubs do not delete events immediately on consumption. In fact they don’t track the consumer state at all. Instead, event hubs use the notion of consumer groups. There is at least one group – the default consumer group, however in many cases you would create more than one consumer group to scale out or add alternate processing behaviors. Each consumer group tracks the location of the reading head – to track the location of the next event. There is one tracking cursor per partition. Event hubs delete messages based on the time-to-live setting on the message. You can supply a time-to-live for a message from one to seven days. Since messages are not automatically removed upon consumption, you can create multiple consumer groups to read and process past messages – a handy feature that lets you analyze a range of messages.
...
Getting Data from Azure – Microsoft Power BI Tool
You can watch the complete demo video.
Summary
Internet of Things is much more than hype. You can leverage the technology to get vital information to drive your business. Companies are investing in IoT in a variety of ways. Some are building better IoT frameworks, protocols, services and systems. Others are looking at providing IoT services or looking at how they can use IoT systems to improve their current solution. A Security related companies will provide security for IoT devices and services, billing companies may evolve to provide solutions for using IoT devices and services, and retail companies – they will know when to deliver a new bottle of milk!
... [Click through for the entire post, the code and more]
Follow @CH9
Follow @coding4fun
Follow @gduncan411
Continue reading...
