In this article by Riccardo Becker, author of the book Learning Azure DocumentDB, we will see how to build a real scenario around an Internet of Things scenario. This scenario will build a basic Internet of Things platform that can help to accelerate building your own.

In this article, we will cover the following:

• Have a look at a fictitious scenario
• Learn how to combine Azure components with DocumentDB
• Demonstrate how to migrate data to DocumentDB

(For more resources related to this topic, see here.)

Introducing an Internet of Things scenario

Before we start exploring different capabilities to support a real-life scenario, we will briefly explain the scenario we will use throughout this article.

IoT, Inc.

IoT, Inc. is a fictitious start-up company that is planning to build solutions in the Internet of Things domain. The first solution they will build is a registration hub, where IoT devices can be registered. These devices can be diverse, ranging from home automation devices up to devices that control traffic lights and street lights. The main use case for this solution is offering the capability for devices to register themselves against a hub. The hub will be built with DocumentDB as its core component and some Web API to expose this functionality. Before devices can register themselves, they need to be whitelisted in order to prevent malicious devices to start registering.

In the following screenshot, we see the high-level design of the registration requirement:



The first version of the solution contains the following components:

• A Web API containing methods to whitelist, register, unregister, and suspend devices
• DocumentDB, containing all the device information including information regarding other Microsoft Azure resources
• Event Hub, a Microsoft Azure asset that enables scalable publish-subscribe mechanism to ingress and egress millions of events per second
• Power BI, Microsoft’s online offering to expose reporting capabilities and the ability to share reports

Obviously, we will focus on the core of the solution which is DocumentDB but it is nice to touch some of the Azure components, as well to see how well they co-operate and how easy it is to set up a demonstration for IoT scenarios. The devices on the left-hand side are chosen randomly and will be mimicked by an emulator written in C#.

The Web API will expose the functionality required to let devices register themselves at the solution and start sending data afterwards (which will be ingested to the Event Hub and reported using Power BI).

Technical requirements

To be able to service potentially millions of devices, it is necessary that registration request from a device is being stored in a separate collection based on the country where the device is located or manufactured.

• Every device is being modeled in the same way, whereas additional metadata can be provided upon registration or afterwards when updating.
• To achieve country-based partitioning, we will create a custom PartitionResolver to achieve this goal.
• To extend the basic security model, we reduce the amount of sensitive information in our configuration files.
• Enhance searching capabilities because we want to service multiple types of devices each with their own metadata and device-specific information. Querying on all the information is desired to support full-text search and enable users to quickly search and find their devices.

Designing the model

Every device is being modeled similar to be able to service multiple types of devices. The device model contains at least the deviceid and a location. Furthermore, the device model contains a dictionary where additional device properties can be stored. The next code snippet shows the device model:

[JsonProperty("id")]
        public string DeviceId { get; set; }
        [JsonProperty("location")]
        public Point Location { get; set; }
        /practically store any metadata information for this device
        [JsonProperty("metadata")]
        public IDictionary<string, object> MetaData { get; set; }

The Location property is of type Microsoft.Azure.Documents.Spatial.Point because we want to run spatial queries later on in this section, for example, getting all the devices within 10 kilometers of a building.

Building a custom partition resolver

To meet the first technical requirement (partition data based on the country), we need to build a custom partition resolver. To be able to build one, we need to implement the IPartitionResolver interface and add some logic. The resolver will take the Location property of the device model and retrieves the country that corresponds with the latitude and longitude provided upon registration.

In the following code snippet, you see the full implementation of the GeographyPartitionResolver class:

public class GeographyPartitionResolver : IPartitionResolver
    {
        private readonly DocumentClient _client;
        private readonly BingMapsHelper _helper;
        private readonly Database _database;
 
        public GeographyPartitionResolver(DocumentClient client, Database database)
        {
            _client = client;
            _database = database;
            _helper = new BingMapsHelper();
        }
        public object GetPartitionKey(object document)
        {
            /get the country for this document
            /document should be of type DeviceModel
            if (document.GetType() == typeof(DeviceModel))
            {
                /get the Location and translate to country
                var country = _helper.GetCountryByLatitudeLongitude(
                    (document as DeviceModel).Location.Position.Latitude,
                    (document as DeviceModel).Location.Position.Longitude);
                return country;
            }
            return String.Empty;
        }
 
        public string ResolveForCreate(object partitionKey)
        {
            /get the country for this partitionkey
            /check if there is a collection for the country found

            var countryCollection = _client.CreateDocumentCollectionQuery(database.SelfLink).
           ToList().Where(cl => cl.Id.Equals(partitionKey.ToString())).FirstOrDefault();
            if (null == countryCollection)
            {
                countryCollection = new DocumentCollection { Id = partitionKey.ToString() };
                countryCollection =
                    _client.CreateDocumentCollectionAsync(_database.SelfLink, countryCollection).Result;
            }
            return countryCollection.SelfLink;
        }
 
        / <summary>
        / Returns a list of collectionlinks for the designated partitionkey (one per country)
        / </summary>
        / <param name="partitionKey"></param>
        / <returns></returns>
        public IEnumerable<string> ResolveForRead(object partitionKey)
        {
            var countryCollection = _client.CreateDocumentCollectionQuery(_database.SelfLink).
            ToList().Where(cl => cl.Id.Equals(partitionKey.ToString())).FirstOrDefault();
 
            return new List<string>
            {
                countryCollection.SelfLink
            };
        }
    }

In order to have the DocumentDB client use this custom PartitionResolver, we need to assign it. The code is as follows:

GeographyPartitionResolver resolver = new GeographyPartitionResolver(docDbClient, _database);
 
docDbClient.PartitionResolvers[_database.SelfLink] = resolver;
/Adding a typical device and have the resolver sort out what /country is involved and whether or not the collection already /exists (and create a collection for the country if needed), use /the next code snippet.
var deviceInAmsterdam = new DeviceModel
            {
                DeviceId = Guid.NewGuid().ToString(),
                Location = new Point(4.8951679, 52.3702157)
            };
 
Document modelAmsDocument = docDbClient.CreateDocumentAsync(_database.SelfLink,
                deviceInAmsterdam).Result;
            /get all the devices in Amsterdam   
        var doc = docDbClient.CreateDocumentQuery<DeviceModel>(
                _database.SelfLink, null, resolver.GetPartitionKey(deviceInAmsterdam));

Now that we have created a country-based PartitionResolver, we can start working on the Web API that exposes the registration method.

Building the Web API

A Web API is an online service that can be used by any clients running any framework that supports the HTTP programming stack. Currently, REST is a way of interacting with APIs so that we will build a REST API. Building a good API should aim for platform independence. A well-designed API should also be able to extend and evolve without affecting existing clients.

First, we need to whitelist the devices that should be able to register themselves against our device registry. The whitelist should at least contain a device ID, a unique identifier for a device that is used to match during the whitelisting process. A good candidate for a device ID is the mac address of the device or some random GUID.

Registering a device

The registration Web API contains a POST method that does the actual registration. First, it creates access to an Event Hub (not explained here) and stores the credentials needed inside the DocumentDB document. The document is then created inside the designated collection (based on the location). To learn more about Event Hubs, please visit  [Route("api/registration")]         [HttpPost]         public async Task<IHttpActionResult> Post([FromBody]DeviceModel value)         {             /add the device to the designated documentDB collection (based on country)             try             { var serviceUri = ServiceBusEnvironment.CreateServiceUri("sb", serviceBusNamespace,                     String.Format("{0}/publishers/{1}", "telemetry", value.DeviceId))                     .ToString()                     .Trim('/');                 var sasToken = SharedAccessSignatureTokenProvider.GetSharedAccessSignature(EventHubKeyName,                     EventHubKey, serviceUri, TimeSpan.FromDays(365 * 100)); / hundred years will do                 /this token can be used by the device to send telemetry                 /this token and the eventhub name will be saved with the metadata of the document to be saved to DocumentDB                 value.MetaData.Add("Namespace", serviceBusNamespace);                 value.MetaData.Add("EventHubName", "telemetry");                 value.MetaData.Add("EventHubToken", sasToken);                 var document = await docDbClient.CreateDocumentAsync(_database.SelfLink, value);                 return Created(document.ContentLocation, value);            }             catch (Exception ex)             {                 return InternalServerError(ex);             }         }

After this registration call, the right credentials on the Event Hub have been created for this specific device. The device is now able to ingress data to the Event Hub and have consumers like Power BI consume the data and present it.

Event Hubs is a highly scalable publish-subscribe event ingestor. It can collect millions of events per second so that you can process and analyze the massive amounts of data produced by your connected devices and applications. Once collected into Event Hubs, you can transform and store the data by using any real-time analytics provider or with batching/storage adapters.

Setting up Azure Search is pretty straightforward and can be done by using the REST API it offers or on the Azure portal. We will set up the Azure Search service through the portal and later on, we will utilize the REST API to start configuring our search service.

We set up the Azure Search service through the Azure portal (http://portal.azure.com). Find the Search service and fill out some information. In the following screenshot, we can see how we have created the free tier for Azure Search:



You can see that we use the Free tier for this scenario and that there are no datasources configured yet. We will do that know by using the REST API.

We will use the REST API, since it offers more insight on how the whole concept works. We use Fiddler to create a new datasource inside our search environment. The following screenshot shows how to use Fiddler to create a datasource and add a DocumentDB collection:



In the Composer window of Fiddler, you can see we need to POST a payload to the Search service we created earlier. The Api-Key is mandatory and also set the content type to be JSON. Inside the body of the request, the connection information to our DocumentDB environment is need and the collection we want to add (in this case, Netherlands).

Now that we have added the collection, it is time to create an Azure Search index. Again, we use Fiddler for this purpose. Since we use the free tier of Azure Search, we can only add five indexes at most. For this scenario, we add an index on ID (device ID), location, and metadata. At the time of writing, Azure Search does not support complex types. Note that the metadata node is represented as a collection of strings.



We could check in the portal to see if the creation of the index was successful. Go to the Search blade and select the Search service we have just created. You can check the indexes part to see whether the index was actually created.



The next step is creating an indexer. An indexer connects the index with the provided data source.



Creating this indexer takes some time. You can check in the portal if the indexing process was successful. We actually find that documents are part of the index now.

If your indexer needs to process thousands of documents, it might take some time for the indexing process to finish. You can check the progress of the indexer using the REST API again.

https://iotinc.search.windows.net/indexers/deviceindexer/status?api-version=2015-02-28

Using this REST call returns the result of the indexing process and indicates if it is still running and also shows if there are any errors. Errors could be caused by documents that do not have the id property available.

The final step involves testing to check whether the indexing works. We will search for a device ID, as shown in the next screenshot:



In the Inspector tab, we can check for the results. It actually returns the correct document also containing the location field. The metadata is missing because complex JSON is not supported (yet) at the time of writing.

Indexing complex JSON types is not supported yet. It is possible to add SQL queries to the data source. We could explicitly add a SELECT statement to surface the properties of the complex JSON we have like metadata or the Point property.

Try adding additional queries to your data source to enable querying complex JSON types.

Now that we have created an Azure Search service that indexes our DocumentDB collection(s), we can build a nice query-as-you-type field on our portal. Try this yourself.

Enhancing security

Microsoft Azure offers a capability to move your secrets away from your application towards Azure Key Vault. Azure Key Vault helps to protect cryptographic keys, secrets, and other information you want to store in a safe place outside your application boundaries (connectionstring are also good candidates). Key Vault can help us to protect the DocumentDB URI and its key.

DocumentDB has no (in-place) encryption feature at the time of writing, although a lot of people already asked for it to be on the roadmap.

Creating and configuring Key Vault

Before we can use Key Vault, we need to create and configure it first. The easiest way to achieve this is by using PowerShell cmdlets. Please visit Command