"Some Sundays start well and end badly"

In order for your device to know which modules it needs to run you need to create a deployment manifest template. It is a JSON file that contains information about modules which a device would run, information about inter-module/cloud communication and some additional settings that you can tinker with.

It is very easy to begin working with a deployment manifest template. It is a very simple concept to understand and you can quickly hop in and develop your application as you see fit. However, the longer you work with it the more bad things you are going to notice.

Messed-up Azure IoT templates

The first issue that I perceived, when working in a 2 man team, was horrible git merge conflicts. Even some small changes resulted in merge conflicts that surrounded more lines of code than one would expect. To add to that, usually when I accepted incoming changes, they had different numbers of opening/closing braces then the original code and I ended up with an invalid deployment manifest afterwards. It was a horrible experience and a time-eater for our project. I don’t even want to think about how git merges would look for a bigger team.

Another issue is duplication of a code when you have several deployment templates inside your project. In our case, we had 3 templates which only had small variations, here and there, with regard to module settings. Every time I wanted to introduce any change, I needed to do it “x” number of times, whereas “x” was the number of templates we currently had in the project. It is very easy to miss replications of changes in one of the templates and for you to lose consistency within those files.

The last thing which was bugging me is the fact that it is fairly difficult to merge 2 different deployment templates. We were working closely with 2 other teams that were developing additional modules for our IoT Edge device. They had fully working deployment templates and it would have been great if we could have just grabbed those templates and quickly merged them into our template. Unfortunately, it was impossible for us to do that. We manually merged incoming modules into our template and, after each update from the other teams, we were required to replicate this process on all 3 of the templates we had.

In light of all the (above-mentioned) shortcomings, it was unproductive to work manually with deployment templates. Subsequently, I planned to create a small library which is going to help eliminate those issues.

Planning is crucial

The first thing I needed to do was to set up the expectations that I had regarding the library that I am going to create. I came up with the following plan:

It should create deployment manifests using python code.

It should save deployment manifests to files and be semantically correct.

If an individual has 2 deployment objects, which have been created by this library, it should be possible to merge them.

Usually, I use C#. This time I chose Python - and this would be my first time creating a Python library. There were several reasons behind this decision, first of all - Python allows creating library code more quickly. It is also simple and fast to integrate a code written in this language with CI-CD pipelines. Additionally, I wanted to learn something new and to publish my first package to PyPi.

Taking matters into my own hands

It was actually a pretty easy and fast process to create the first version of the library and to upload it to TestPyPi — the test version of PyPi which allows you to play around with and publish your pre-release packages. It is also very easy to push packages there and I had only studied the official documentation in order to learn how to use it.

Initially, I was a little afraid of dealing with package publishing, as I had never done it in Python. My fear quickly disappeared, as the documentation is pretty solid and makes it fairly easy to setup your library in order to publish it to TestPyPi/PyPi.

The practice which I find very helpful, when creating a new library/module/service, is to first create a file with sample syntax that my code interface would provide. This helps me to prototype my code and create a solid understanding of how each component I am going to create will be connected with others. It is easier to change an approach during this stage, rather than to later discover that my interfaces are going to create a mangled “code spaghetti” which would be hard to maintain. This time it was pretty simple ([LINK]), however this approach scales pretty well even with more complex code. (You can easily see that I had not touched the python code in a long time and did not know that one does not need to insert the word new when creating new objects).

After the package was published I could use it on my local environment, see it in action as well as how it works. It worked, however, it had bugs. Oh boy. Actually, a lot of bugs.

I started debugging the issues and I quickly overcame all hurdles. I had no automated tests and needed to test everything manually. It was pretty fast to work in that way, as I was working in a constant change-save-run-check loop. Python runs my local test code instantaneously, so it was quick to work in that mode.

The Result

After fixing all those issues I can now present to you IoT Deployment Generator or IDG in short. It achieves all the goals I had set earlier and the code approach proved to be a great replacement for tweaking manifests manually. Goodbye horrible merge requests and welcome concise setup code!

I think it is time to demonstrate some sample code to show you how the library can be used in a specific scenario. I am going to show you the code that can recreate one of my first deployment manifests.

The following code [CLICK!] creates semantically the same file as the deployment template I mentioned earlier. It is more concise as it does not have the huge amount of boilerplate code you would usually need. It is easier to manage as you can split deployment generation code across multiple files. It is more pleasant to work with, because git merge conflicts are simple when dealing with a code. It is better.

Additionally, the code is 3 times shorter when compared with the original template manifest.

Future maintenance of the repository

I plan to maintain my library for as long as possible. I still have some work to do in order to make that library better. Next step would be to plan some CI-CD, tests and additional features that could benefit other users. Additionally, not all settings are tweak-able and I am not entirely sure if there is a reason for them to be changed.

If you are interested in helping me to further develop this library then I will be happy to welcome you aboard!

Honorable mentions

There is a library from Microsoft that supposedly should have similar functionality to IDG, however, from the readme I was not sure whether or not it was going to satisfy my requirements when it came to template manifest creation and maintenance.

This library helped me to deliver high quality code to my client as well as improve their internal processes involving IoT Edge. This adheres to principles of Xfaang to provide excellent software quality for our clients to reach their tech dream.

Visit https://xfaang.com/ to learn more.

Tags: