Site icon 지락문화예술공작단

Internet of Things on the Xbox (App Dev on Xbox series)

KENNETH

Internet of Things on the Xbox (App Dev on Xbox series)

This week’s app is all about the Internet of Things. Best For You is a sample fitness UWP app focused on collecting data from a fictional IoT enabled yoga wear and presenting it to the user in a meaningful and helpful way on all of their devices to track health and progress of exercise. In this post, we will be focusing on the IoT side of the Universal Windows Platform as well as Azure IoT Hub and how they work together to create an end-to-end IoT solution. The source code for the application is available on GitHub right now so make sure to check it out.

If you missed the previous blog post on Hosted Web Apps, make sure to check it out for in-depth on how to build hosted web experiences that take advantage of native platform functionality and different input modalities across UWP and other native platforms. To read the other blog posts and watch the recordings from the App Dev on Xbox live event that started it all, visit the App Dev on Xbox landing page.

Windows IoT Core

IoT, or the “Internet of Things,” is a system of physical objects capable of sensing the internal or external environment, connected to a larger network through which they are sending data to be processed and analyzed, and finally synthesized on the application level. This is intentionally a broad description, as IoT can take many shapes. It is the smart thermostat in your house, a water meter system on a massive hydroelectric dam, or a swarm of weather balloons with cellular data connections and GPS sensors.

The goal of most IoT scenarios is similar; gain a specific insight from, or operate on, its environment. For the purposes of this article, we’ll use an example of the smaller IoT systems that have the responsibility to collect a specific set of data using sensors and send that data to a more powerful system that can gain intelligence from that data to make larger decisions. Later in the post, we’ll reveal the fictional smart yoga ware and see how we can use Windows IoT to power the gear.

Windows IoT Core is a version of Windows 10 that is optimized for smaller devices with or without a display; devices such as Raspberry Pi 2 and 3, Arrow DragonBoard 410c, MinnowBoard MAX and upcoming support for the Intel Joule. There is also a professional version, Windows IoT Core Pro, that adds many enterprise friendly features, such as:

To install Windows IoT Core on a device is easier than it has ever been. You can use the Windows IoT Core Dashboard tool that automates the process of downloading the correct image for your device and flashing the OS onto the device’s memory for you.

Windows IoT leverages the flexible and powerful Universal Windows Platform. Yes, this means you can use your existing UWP skills, including XAML/C#, and deploy almost any UWP app onto an IoT device providing that you’re not leveraging special PC hardware (e.g. a AAA game that requires a powerful graphics card). Majority of UWP APIs work the same way, but you can also get access to IoT specific APIs on Windows IoT Core by simply adding a reference to the Windows IoT Extensions for the UWP. Getting started is really easy, and once the reference has been added you’ll get access to namespaces like Windows.Devices.Gpio and Windows.Devices.I2C (and many more) to begin developing for IoT specific scenarios.

Deploying a UWP app to an IoT Core device is the same as deploying to any remote Windows 10 device; no special knowledge set is required for this either! Simply select Remote Device as your target and put in the IP address (or machine name) and start debugging. Take a look at the Hello World sample app tutorial for Windows IoT to see just how easy it is.

Best For You

Let’s continue with an idea where we have invented smart yoga pants. A small IoT device is embedded in the Best For You yoga pants running Windows IoT Core with sensors woven into the fabric to capture environmental data such as heart rate, temperature (temp sensor) and leg position (flex sensor). These sensors are very small and virtually undetectable in the pants. The app running on the device constantly captures the incoming data from the sensors and sends it to the cloud for further processing.

Remember that the IoT device’s responsibility in this scenario is to monitor and report, not process the data. We’ll leave the processing up to more capable machines with much more processing power. There are many ways for the device to transfer this data, such as the very convenient and traditional HTTP (if the device can be connected to the internet directly), Bluetooth to a mobile device that can relay the data, and even through an AllJoyn (or other wireless standard) connection to a device like the IoTivity AllJoyn Device System Bridge.

With a way to communicate the data, where can we send that data so that it can be processed into meaningful insights? This is where Azure IoT Hub is ideal.

Azure IoT Hub

Azure IoT Hub is a powerful tool that allows for easy connection to all your Windows IoT devices in the field. It is a fully managed service that enables reliable and secure bi-directional communications between millions of Internet of Things (IoT) devices and a solution back end.

Here’s a high level architectural diagram of a Windows IoT Core solution with Azure IoT Hub and connected services to visualize and process the data:

You can provision your Windows IoT devices so that they’re authenticated and can connect directly to the Hub. Provisioning your IoT device for Azure Hub is also easier than ever, the same IoT Dashboard you used to install Windows IoT lets you provision devices with Azure IoT Hub. You can read more about it in this blog post; here’s a screenshot of the IoT Core Dashboard’s provisioning tool:

The Hub receives communications from the devices that contain data relevant to that’s devices’ responsibility. This usually takes the form of little data packets for each sensor reading. To continue with our smart yoga pants example, the IoT device’s UWP app takes a reading from all the sensors every second.

Capturing our sensor reading and sending it Azure IoT Hub might look something like this:

while (true)


{
    var pantsSensorDataPoint = new
    {
		rightLegAngle: rightLegSensor?.Value,
		leftLegAngle: leftLegSensor?.Value,
		bodyTemperature: tempSensor?.Value
        };

        var messageString = JsonConvert.SerializeObject(pantsSensorDataPoint );
        var message = new Message(Encoding.ASCII.GetBytes(messageString));

        await myAzureDeviceClient.SendEventAsync(message);

        Task.Delay(1000).Wait();
}

As we can see, the three sensors report a particular value, that we want to send to the Azure IoT Hub for processing.  Notice the myAzureDeviceClient; this is a DeviceClient class that comes from the Azure IoT Hub SDK (adding the SDK to your app is as simple as adding the Microsoft.Azure.Devices nuget package).

There is a little configuration to instantiate the client with your IoT Hub’s details, but when that’s ready all you needs to do to send up some data is call SendEventAsync(). To learn more about setting up the hub, check out this Getting Started with Azure IoT Hub tutorial, it covers everything you need to get up and running quickly. It simulates the IoT device with a small Console app, but you can replace that with your Windows IoT Core device’s UWP app as the nuget package can be added to UWP app as well. There is also a great Visual Studio Extension available to help you get configured quickly.

Alternatively, you can use an ARM template (ARM = Azure Resource Manager). An ARM template allows you to do an amazing “one-click deploy to Azure”. The template has a json file that defines the resources and the connection between those resources. An example of this is the ARM template linked in the readme of the project in GitHub.

Okay, so now we have the IoT Core device sending data to the Azure IoT Hub every second. How can we make use of this? How do we get insightful information from so much data? Let’s take a look at how we present the data.

Presenting the data

Once the data is being stored by the Azure IoT Hub, it can be used by other applications or used directly for analytics. We’ll cover two scenarios for the yoga pants data: Streaming analytics and a client UWP app running on an Xbox One!

Stream Analytics

You have the ability to hook up an Azure Streaming Analytics job to your Azure IoT Hub. The database that your Hub stores that data to becomes a treasure trove of information that can be plugged into a service like Power BI to be molded into insightful charts and graphs that present the information in a meaningful way.

First you need to setup your Stream analytics job, once that’s prepared, you create a query against the database using Stream Analytics Query Language (SAQL), this is very similar to SQL queries. An example to query against the yoga pants data to might look like this because it only has three relevant fields; LeftLegAngle, RightLegAngle and BodyTemp.

SELECT * FROM YogaPantsSensorTable

The output from the Stream Analytics would have each reading from every user. This is also known as a “passthrough query” because it sends all the data through to whatever consumes it. You can also take a look at other examples of how to use the query in this tutorial:  Get started using Azure Stream Analytics: Real-time fraud detection.

We could now connect the Stream Analytic Job to a service like Power BI in order to show the data in a multitude of charts to get at-a-glance information from all your sensors and users. Stream Analytics can support millions of events a second. This means you could have smart yoga pants for everyone, across the world, in a special world-wide yoga session and get immediate telemetry streaming into your data visualization apps. For more information on how to use Power BI and Stream analytics check out this tutorial: Stream Analytics & Power BI: A real-time analytics dashboard for streaming data.

Presenting Data in a UWP App

Now for the UI magic that bring all this together for the consumer’s delightful user experience. We’ll want a UWP app that runs on Xbox One (keep in mind that because this a UWP app, we can also run it on PC, Mobile and Hololens!).  Let’s start focusing on the Best For You demo app and how it delivers the experience to the user.

First we need to step back and think about some design considerations. When designing an IoT app for any device, it is crucial to think about the context in which the end user will be experiencing the app. A classic example of this train of thought would be if you were designing a remote control app for an IoT robot. Since the user may need to walk around during this interaction, the targeted device would be a phone or tablet.

However, when designing Best For You, we decided that the Xbox One is perfect for an exercise-focused app. Here are just a few reason why:

Now that we know what the target device will be and how to design for it, we can start thinking about the flow of data from IoT Hub to the UI. The smart yoga pants have embedded heart rate sensors and has been sending this data to the Hub and we want to show this in the UI.

Let’s take a look at how the Best For You app connects to the Azure IoT Hub and gets the user’s heart rate. The demo app has a StatsPage.xaml, within that page is a StackPanel for displaying the user’s heart rate:


<TextBlock Text="{x:Bind HeartRate, Mode=OneWay}" />

This TextBlock’s Text value is bound to a HeartRate property in the code-behind:


public string HeartRate
{
    get { return _heartRate; }
    set
    {
        _heartRate = value;
        …
        RaisePropertyChanged();   
    }
}

Now that the Property and the UI are configured, we can start getting some data from the Azure IoT Hub and update the HeartRate value.

We do this within a Task named CheckHeartRate, let’s break the task down. First, we need to connect to the IoT Hub:


var factory = MessagingFactory.CreateFromConnectionString(ConnectionString);

var client = factory.CreateEventHubClient(EventHubEntity);
var group = client.GetDefaultConsumerGroup();

var startingDateTimeUtc = DateTime.Now;

var receiver = group.CreateReceiver(PartitionId, startingDateTimeUtc);

The EventHubReceiver is where we can start receiving messages from the IoT Hub! Let’s look at how to get data from the EventHubReceiver:


while (true)
{
    EventData data = receiver.Receive();
    if (data == null) continue;

    await Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () =>
    {
        HeartRate = Encoding.UTF8.GetString(data.GetBytes());
    });
}

That’s it! Calling the Receive method on the EventHubReceiver and will get you an EventData object. From there you call GetBytes and since out HeartRate property is a string, we convert it to a string and then update the HeartRate property. The data has now made the successful trip from pants sensor, to Windows IoT, to Azure IoT Hub and finally to UWP app on Xbox One!

Here’s what the UWP’s app UI looks like on Xbox One (see the heart logo and the heart rate at the top right):

This isn’t necessarily the end of the journey of the data from sensor to UWP app. You can add a sharing mechanism into the UWP app and have the user share their progress and scores on social media and engage other users of your amazing smart yoga pants solution. Alternatively, you could also gamify the app and have a leaderboard in Azure and use Stream Analytics pull in the currently trending users.

That’s all!

Now that you are here, make sure to check out the app source for the UWP app on our official GitHub repository. Read through some of the resources provided below in the Resources section, watch the event if you missed it and let us know what you think through the comments below or on twitter.

Don’t forget to check back in next week for yet another blog post and a new app sample where we will focus on how to take advantage of the camera APIs in UWP and how to add intelligence by using the vision, face and emotion APIs from cognitive services.

Until then, happy coding!

Resources

Previous Xbox Series Posts

Exit mobile version