GET IN TOUCH WITH PAKKO, CREATIVE DIRECTOR ALIGNED FOR THE FUTURE OF CREATIVITY.
PAKKO@PAKKO.ORG

LA | DUBAI | NY | CDMX

PLAY PC GAMES? ADD ME AS A FRIEND ON STEAM

 


Back to Top

NUCL3AR // DIGITAL ARTS

Augmented Reality and IoT project using Adafruit IO - Hackster.io

Augmented Reality and IoT project using Adafruit IO – Hackster.io

This tutorial is thought to be the ending point where I aim to build an Augmented Reality & IoT program.

At this stage I’ll take data from a MQTT broker (Adafruit IO io.adafruit.com) to show it in an Android App built using Unity and Vuforia.

You can check my previous projects to learn how to send data to a MQTT broker and how to show that MQTT data in Unity.

For this stage I’m taking as reference the following YT video that I find very helpful

I’ve chosen Unity because it is one of the most used AR softwares, because it has mqtt libraries developed and because it can interact with Vuforia.

Adafruit IO is the MQTT broker that receives information from an IoT device.

Unity is the platform that will take the information from Adafruit IO and show it in a digital environment. It will also help us to build the App that we will run in our mobiles or tablets.

Vuforia is the the software that recognizes and tracks target images in a digital environment.

Therefore, the project at is core is run using Unity that connects to Adafruit IO and uses Vuforia functionalities.

You’ll need the following:

First create an account to Vuforia (https://developer.vuforia.com). Then continue by downloading the latest version of Vuforia to Unity package from https://developer.vuforia.com/downloads/sdk

Log in into your Vuforia account and click on Develop, Target Manager and Add database. Give it a representative name for your project and select the Device option to create the Database.

The Database will store every target image that you will have in your project.

Now, select the recently created Database and then on Add Target.

In our case the target will be a Single Image, so select that option and then select an image file from the browser. In my case I selected a QR code that I created for this project. Assign a representative name for that image that will later be used in Unity. Once that you have added the target images, click on Download Database and then in Database for Unity Editor.

Last but not least, still in Vuforia webpage click on License Manager, assign a name to your project, check the box and confirm to get your License Key. That Key will allow us to run Vuforia functionalities on Unity.

That’s it, you’ve already have all the documents and the License Key that you need from Vuforia.

Once that you have downloaded Unity, install the software and run it. If you are going to run a mobile or tablet app you should also install its extensions in this step. In my case I installed the extensions for an Android App.

In the main page of UnityHub select New Project, then 3D Core option and assign a representative name to it. In this case, I named it as I named the Vuforia one “AR_IoT_QR”

Drag and drop M2Mqtt and M2MqttUnity folders downloaded from Github to the Assets of the project as shown in the image below.

Then, create two new folders inside the Assets file where you will store the main scripts that will run your program. In my case I called those folders CE and Scripts

Inside of Scripts drag and drop mqttReceiver.cs and mqttController.cs files. You can find both scripts attached with this project.

In left side of the screen click the right button and create an empty Game Object, name it MQTT_Receiver and add the script mqttReceiver.cs dragging the script into it. Then, complete the fields on the right side of the screen.

Create a new GameObject, name it ControlledObject and drag and drop mqttController into it. Then click on Event Sender tab and Select MQTT_Receiver.

Lastlly, create a 3D Object – Text Mesh Pro and drag and drop mqttController.cs into it.

In this case you’ll need to edit the script by clicking in the three vertical dots at the right of Mqtt Controller as we did in the previous stage. Erase the whole content of the scrip and paste the following:

Now go to your Downloads folder and open Vuforia to Unity package. Once that is installed, create an AR camera by clicking on the plus symbol, Vuforia Engine and AR camera as shown in the image below.

Once that you’ve created an AR camera object, now right click over the AR camera object and select Vuforia Engine and Image Target. Continue by opening your projects database that you’ve also downloaded from Vuforia and importing it to Unity. Finally, select AR camera object and in the right side of your screen click on Open Vuforia Engine configuration to paste the licence Key in the Add license Key box as shown in the image below.

Lastly, go to Image target, select Image Target Behaviour, type From Database, select the respective project and the target image that you want to use as target.

To display the image into Unity simply double click on Image Target Object.

You can set different image views by playing with the box and the coordinates at the right top corner of the image.

We’ve already set the structure where the software will track the target image.

To design the digital environment we will start by scaling the target image to X=50, Y=50 and Z=50. We are doing this for practical reasons during the design, once that we’ve finished with the settings we then need to set the scale back to 1.

Now we are going to create a dashboard where Unity will display the desired variables to control. I designed one to display temperature and time values. You can find the image attached with this project.

Right click on Image Target Object and selecting 3D Object and then Plane

Then drag and drop the dashboard image into Resources and then drag and drop it again but this time to the Plane Object

You can then manipulate the image as you like, in my project I liked how it looked like emerging from the image so I needed to rotate it 180 degrees in the y-axis and 90 degrees in the x-axis (but in a later step).

To modify the size of the dashboard we simply click on the Rect tool (square button) and change the size to an according one taking as reference the QR code. You can also change the location of the dashboard image by moving the image using the Move tool (arrows button)

After a few relocations I finished with the image as I wanted.

Now we will continue by locating the Text (TMP) objects that we want to display in the Dashboard image.

Once that you’ve located them drag them into the Image Target Object, this will help you to re-size everything in one shot.

One important thing that I noticed is that if you are aiming to display more than one variable at the same time you’ll have to create every variable from scratch. Do not copy andpaste an existing variable to create a new one because the program will only display the original variable and it won’t display the copy-pasteones.

Once that you’ve set the digital environment re-scale the Image Target to X=1, Y=1 and Z=1 and save the project.

If you press PLAY button and then Scene you should see how the Text (TMP) Objects start showing your MQTT inputs.

You are almost there! The last step is creating an Android App. To do so, go to File and then to Build Settings.

Then select the OS where you are going to run your app (you should see Unity logo at its side!) and finally press Build. In my case I’ve chosen Android. Select a place to save the.apk file. This is the file that you are going to transfer to your phone to run your app.

While building your project you might come up with this building error.

To solve it you need to change the Target API from Target API Level field in the Player Settings window. You can find the Target API Level scrolling down until you find Identification in Other Settings. You’ll need to select the same Android version according that your mobile/tablet has.

After doing that modification you should be able to build your app without issues.

Finally, you’ll need to plug-in your device and drag and drop the.apk file into the Downloads folder. Now take your device, open Downloads folder and select the.apk file to install your app. Once that it has been installed you should see the app in your main Menu.

If you open that app probably your OS will ask you if you approve opening an app created by an unknown company, you should approve that and if later asks you for a Play Protect analysis you should skip it.

That’s it you’ve created your Unity and Vuforia digital environment and you’ve created your own app.

You are now ready to start using your Augmented Reality and IoT app.

This content was originally published here.