Behind technology: What seperates the winners from the also-rans

Hello Internet, in today's article we shall why in order to build long lasting solutions, we should keep aside the eternal arguments about what technology to use for building software and rather bring out a large whiteboard and first attempt to solve problems in a time tested way: brainstorming, diagrams and formulae.

Lasting companies and solutions: What seperates the best from the good

Let's play a small rapid fire game. Try to answer the following questions within seconds of reading.

1. You need to search for the recipe of a truffle cake. Which site will you head for?
2. You are standing at the airport and need a taxi to take you home. Which app do you reach for?
3. You have some awesome pictures from your visit to Thailand. Where do you upload your photos?

Chances are that you didn't think even for a second before answering. In fact, the answers might have hit your mind even before you read the question completely.
What makes solutions like Google search, YouTube, Uber, Facebook, Picasa, Flickr and Instagram so good at their game? It is very easy to conclude that technology is the ONE answer to such seamless solutions. However, technology is only the means to an end. The same technologies that are available to the top 10 apps / websites are also available to everyone else. In the great levelling playing field called the Internet, easily downloadable and commoditised technology is not what seperates the best from the good. These companies are in fact excelling at something that happens even before technology is applied, or even before the engineers finalise on what technology is best for the situation. It seems very contradictory, but these companies are good at sitting down together in front of a whiteboard, writing down points on a marker, drawing diagrams and paper models and playing out the entire solution with paper and pencil before they enter the technological world of bits, bytes, codes, apps, SaaS and the market.

  

Image courtesy: 123rf.com

But just what is it that they are drawing on whiteboards and modelling on paper before sitting down to write programs? And how does that make a difference to the final solution? As it turns out, they are figuring out something very fundamental to the world of computing. Regardless of whether you use Android, iOS, Java, Python or NodeJS, computing devices will understand only two building blocks. Enter data structures and algorithms.

Data structures and algorithms: What they are

Computing devices are made to store data over a long period of time. Since data storage is such a core functionality in computing, scientists and engineers have turned extensively to mathematics to study the various forms in which data can be represented, such as sets, lists, matrices and the like. Different types of data need to be stored in different forms and there is an entire subject behind the study of how efficiently to store data such that it can be accessed fast when required later. The form in which computing devices store data is called data structure.

In order to store data and fetch it for use later, a computer must go through a set of instructions which when given a piece of data must find the right place inside to data structure to put it and then later use another set of instructions to fetch data quickly from the structure again. Just like data structures, a carefully planned sequence of instructions goes a long way in ensuring that there are no major bottlenecks while retrieving data and that the retrieval is as quick as possible. Such as sequence of instructions is called algorithm.

A well planned combination of data structures and algorithms can mean the difference between a smooth and a frustrating experience for the user. Planning a good combination often requires engineers to abandon a computer for a week or several weeks and simply draw out diagrams and write formulae on a whiteboard or a piece of paper before writing the first line of code.

Using data structures and algorithms: A real life example

To really understand how the choice of data structures and algorithms affects user experience, let us take a real life example.

1. Let us say that Eddie has just moved into a new neighbourhood and is invited over to a get-together in his neighbourhood's club. Eddie takes along a sheet of paper and a pen, meets new people and notes down their names and numbers. He notes down names in the order in which he meets the persons and the names are not sorted in any order which makes sense to Eddie when he wants to look up specific names later. However a list works for him right now, since noting down is best done in the simplest way possible. This way, even if he meets three persons together and all of them rapidly dictate their names and numbers to him really fast, he can keep up, since he is simply appending new names to the list and his hand can instinctively reach the end of the list as soon as he hears a new name and number.
2. However the list is not sorted in any particular order that makes sense, so Eddie goes home and fishes out his phone book. The phone book is structured such that it is easy to find names in a snap. The names are arranged in alphabetic order, with pages having markers for alphabets. To find a name, Eddie just goes to the page which is marked with the alphabet with which the name of the contact he is seeking starts. E.g. to find the number of a friend named Carlos, Eddie simply goes to the page marked with letter C, on which he can find all his contacts with names starting with C. Carlos will be right up there.
   

   

   Image courtesy: amazon.com
3. Like any good affable person, Eddie wants to wish his friends on their birthdays and anniversaries. Let's say the day is 5th March. Alas, the phone book has no good way to pick out if any of his contacts has an event going for him / her that day. Even if he records each contact's birthday against his / her name, combing through each contact's record to find out if this day is special for him / her isn't a very attractive option. But Eddie is smart. He has a good calendar, where he marks the name of the person and the occasion related to the that person, against that date on the calendar. Eddie just looks up the date on the calendar and calls the person to make his / her day.
   

   

   Image courtesy: weknowyourdreams.com
4. It is the soccer world cup final and Eddie wants to invite over all his soccer loving friends to his home to watch the match on his big screen smart TV at night. However, not all his friends are into sports and not everyone likes to stay up at night to watch TV. Again, going through his phone book is not the best option to gather a list of people who like sports and do not mind staying up late. Luckily, Eddie has been fairly inquisitive while talking to his friends and has smartly made categorised lists on who likes soccer and who are night owls. He just needs to look at his categorised lists to find friends who would fit the occasion. He can look up the names under the categories and then use the phone book to find the phone number and then call each friend.
   

   

   Image courtesy: lifehack.org
5. Eddie decides to throw a barbecue party at his home's lawn. To make it personal, he decides to visit each house to invite his friends in person. He decides to bike around the neighbourhood and distribute hand drawn invitation cards to each family. But what route should he take to cover each family's home and do it in the shortest route possible? Will his phone book or calendar help here? Of course not. Perhaps a map of the neighbourhood will. And Eddie does have one. So he sets down to work, drawing a good route with a pencil over the map. Before long, the cards have been distributed and Eddie is set to have an awesome barbecue party.
   

   

   Image courtesy: 123rf.com

As you can see, Eddie has been paying attention to how he is going to use his contact list for different purposes and has prepared himself for these. By augmenting his contact list with a phone book, calendar, categorised lists and a map, he has given himself a very good chance of smoothly getting exactly the information he needs about his neighbourhood instantly. Had he missed out on any of the tools mentioned, it would have been a pain for him to get the right information. The different tools he used were identical to data structures and methods he used to retrieve the info are algorithms, e.g. turn to letter C on the phone book, look up 5th March on the calendar, draw a route on the map, etc.

The above illustration shows how having different representations of the same small neighbourhood and its people can drastically improve or destroy the experience of having to use that information. Imagine what it would be for Google or Uber to not use the right data structures and algorithms for the right purpose, with the humongous amount of data that they have.

Conclusion

Using the right data structures and algorithms seems to be common sense, but more too often, companies get caught up in the lure of technology and fail to pay enough attention to the basic building blocks and the core foundations of data structures and algorithms. Most companies in fact brag about having used cutting edge technologies such as their solutions being built on NoSQL, using web sockets or using cloud computing and we have very few companies that boast their algorithm can allow them to service 10000 users at once.

Since the inception of computing, data structures and algorithms have been unsung heroes and they still continue to be unsung, but that does not change the fact that they are heroes.

Measuring and Charting the progress

Hello Internet. We have seen how devices gather data from the field and send it over to the enormous cloud for storage. However stored data is of little use for a person who is trying to gather meaningful insight from it. The data sent over is raw data and is very difficult for the ordinary user to make sense. This is where querying and reporting comes in. Extraction of volumes of historic data and presenting them in the form of reports is what puts value into the painstaking collected data.
Here is a pyramid which shows data and how it is distilled into wisdom.

The basic strategy to convert huge data into wisdom: Divide and Conquer
Because of the sheer volume of data, it is smart to first break down the gathered data into manageable pieces and then convert those pieces into something meaningful. A single system generally has no means to crunch all the data in one bite on its own. Here are some of the strategies and technologies used to achieve conversion.

Crunch the data as it comes

This is the most preferred strategy when it comes to real time reports. The data that is coming in is crunched live into different types of outputs. Since the data generally comes in really rapidly (also called firehose input), the crunching system must have a high amount of processing power. Each input is generally processed into fifty or hundred types of real time metric, such a number of users currently online, number of active transactions, live incremental count of progress, etc. Also since these metrics are written live, it is a big NO-NO to store such rapidly changing data into disk storage such as hard disk. The live data must reside in faster memory such as solid state device or RAM. Multiple copies and backups are desired since such types of memory devices are typically temporary and lose the data when electricity is stopped or if the memory happens to be disconnected. Apart from hardware, it is necessary to use software which must be capable of accessing data & databases in-memory rather than from the disk. Traditional databases such as Oracle & MySQL do not have a good support for memory databases due to their heavy memory footprint and overhead. Enter Redis. Redis is one of the most successful in-memory data storage mechanisms and can support voluminous amounts of reads/writes per second. Other solutions such as Cassandra can make databases both on disk & in-memory.

While making real time metrics, it is important to plan out the metric well in advance since it would be very difficult to make a new metric on the fly due to the amount of crunching needed to bring the data upto date and then keep up with real time.
Another important point with real time data is that it must be simple enough to be derived in a very short time. E.g. Total online users at the moment is simpler to derive than total online users who are between 26-30 years of age residing in London. Not only does such conditional derived metric need more processing, but one would then have to make real time metrics for each such age group & city, which will quickly eat up the memory. Such reports are better off derived periodically and stored on a higher volume storage such as a disk to be fetched later.

Crunch data periodically

For periodic reports such as daily reports or monthly reports, this is the most efficient strategy. Since such reports are not required at real time, one can schedule them to be generated at intervals when the traffic is either very low or non-existent, such as midnight every day. Periodic data are generally summaries and do not need to store each & every field that is in the main transactional databases. For this reason, they are kept seperate from the transactional databases and even use different technologies such as hierarchy databases to show metrics grouped under various categories. The most common databases for periodic reports are MongoDB and Cassandra. The data is generally stored on disk or on a long term storage such as Amazon EBS, S3 or Glacier, which are cloud storage offerings, so that the business does not need to maintain its own storage disks for storing historic data.

Ad hoc reporting: Use multiple processing units to process blocks of data parallely

Despite all the careful planning, there will be instances when a business will need to generate reports based on adhoc requests. In such cases, it may be needed to call upon the business's entire history of data since the day it started operations. Such reporting is daunting and the business should ask for a reasonable amount of time to return the results to the stakeholder which raised the request. In the meantime, the business can use the following methodology to generate the report. Divide the data into manageable pieces. Set up a sizeable number of processing units to work on each piece. Start parallely processing each piece to gain outputs per piece. Gather the piecemeal output from each processor and join them up to get the final result. The above steps have one name in the industry: Map-Reduce. Mapping is the process of dividing and assigning pieces to processing units, whereas reduction is the process of gathering piecemeal results and combining them to get one result. The enormous data from which to obtain the result is called Big Data. The most popular technology used for Map-Reduce is currently Hadoop. Other company offering such Google Big Query and Amazon Elastic MapReduce are also gaining traction.

Over the past few blogs, we have seen how IoT works from the sensor data gathering phase to generating reports on the cloud. From the next post, we will see popular posts related to IoT, that I will curate from over the Internet.

Gathering and recording data from the field

We have seen in the previous blogs how devices on the field shoot out data to the cloud. We have also seen how the cloud can instruct the field to carry out remote instructions. We will now see what one uses on the cloud in order to set up the infrastructure to manage huge reams of field data.

What exactly is the cloud

Jargon speak has introduced a lot of words these days and the word 'cloud' is even more prevalent than 'IoT'. To super-simplify the definition, think of it this way. Instead of setting up our own infrastructure which would need 24x7 electricity, Internet connection, software and hardware maintenance and thus a humungous effort and a support team, we hand off all the headache to a provider who takes care of all these. This provider will make sure that he offers the following features:

• Server machines where our services will be hosted. The biggest feature is that server machines can be added quickly on the fly if the service's loads are taking a big hit during peak hours.
• Basic or complete suite of software which is needed for our services. Most providers will provide plans such as web hosting package or Web/Database combo package. Some of them allow customization to allow us to install our own software. Almost all of them have plans which set up the basics and then offer us a way to access the machines so that we can tweak them as we see fit.
• 24x7 availability with support. While a 100% availability is not guaranteed, most services go for 99 or 99.9% uptime.
• Payment plans which are generally very economical. Most of these services offer pay-as-much-as-you-use model. Almost all of them offer an advance lock-in for a few months or years at much cheaper rates if we commit to hosting our services on them for a long term. E.g. if the service costs 3 cents per hour of uptime, a service provide may agree at 2 cents per hour if we commit to hosting with them for the next 6 months.
• Dashboards to control, monitor and oversee the performance and tweak settings and add/remove servers on the fly.

The most popular cloud services at the moment are Amazon & Rackspace. Both of them offer Windows and Linux servers. Google App Engine and Google Cloud Computing have made huge inroads in the last 2 years with their super high performing and highly standards compliant offerings, but their solutions are Linux-only and hence there is no support for .NET based applications. Microsoft on the other hand offers the Windows-only Azure.

Image Courtesy: rackwareinc.com

Image Courtesy: digitalmunition.me

Image courtesy: cloud.google.com

Image courtesy: cloudtimes.org

This is by no means a complete list and there are many local and regional players in the game, such as Netmagic here in India. In Europe and USA, almost every Internet Service Provider also doubles up as a cloud provider.

What comes across from the field to the cloud

At the moment, there is no standardisation in the protocol or format with which data can be sent over from the field to the cloud. HTTP is the de facto protocol of choice since it has been around for far longer than the other protocols. Tried and tested, it is easy to get the man power to churn out a HTTP based data transfer service, since the World Wide Web uses the same protocol. However it is not uncommon to see other solutions such as Jabber (the most common chat protocol) or MQTT (the latest IoT specific data transfer protocol).

As far as the format of data goes, different vendors use different ways to represent the data. JSON is the most common format, followed by XML. However, many solutions use their own proprietary format, specially tailored for their IoT system and not interoperable with any of the others.

JSON and XML are particularly popular because of the ability to represent one piece of data inside another and thus achieve a hierarchy and an order of structure, which is how objects exist in the real world. However JSON has recently gained more popularity over XML since JSON can represent the same data in less number of bytes and is hence seen as network friendly for slow GPRS based networks. Some companies stick to XML though, since data represented in XML can be validated before being accepted on the server using a standard technique called DTD, without writing a line of software. No such standard standard validation technique exists for JSON and companies much implement their own validation mechanism by writing their own code / algorithm.

Typical JSON data showing employee records

Typical XML data showing book records

While JSON is receiving rapid adoption, there is no common JSON structure that is standardized across two vendors, thus defeating the purpose of JSON's interoperability across systems. E.g. In the employee structure above, one could split name as 'firstName' and 'lastName' by another solution. A system which is programmed to read 'name' would fail to recognise the split names.
To solve this, IoT has been increasingly tapping into a new standard called schema.org. This new standard has defined the JSON structure for many common type of 'things' such as an employee. Conforming to these pre-standardised structures will allow systems from various vendors to interoperate and synergistically work together.

Schema.org JSON structure standardised for a store. Any software which needs to represent a store should ideally follow this structure and that sofware will find it easy to work with stores all over the Internet.

How the cloud stores data

The data that comes from the field is fleeting and needs to be recorded somewhere for further analysis. There are several methods available to record and store the data. The obvious choice to do so is in databases. But there are so many types and brands of databases available that must be evaluated, compared and selected to arrive upon the best solution.

Broadly there are three main types of databases available these days: Traditional relational databases which have been around since the Darwinian days of digitally recorded data, hierarchical databases (typified by the term named 'NoSQL' these days) and linked databases (also called Graph).

Relational Databases

Avoiding a highly technical definition, let's just say that in relational databases, data is stored in tabular rows and columns, each table representing an entity or a 'thing', such as an employee. One table can link to another related table using a method called 'joining'. E.g. If there is one table storing all employee details such as his personal and contact details and another which stores his/her income and tax details, then for making a tax statement which shows the employee name and his/her tax liabilities, the two tables must be joined to fetch the info and make the report. Joining is computationally expensive and has been known to slow down data fetching operations. While relational databases are excellent for crunching transactional records into summaries very quickly, fetching data from multiple tables for the purpose of reporting is a massive bottleneck.

The most popular database solutions in the market currently are Oracle and MySQL. Microsoft based systems use Microsoft SQL server.

Hierarchical Databases

Borrowing from the concept of JSON described above, hierarchical databases use JSON structure to nest different types of data into each other, thus eliminating joins. Using hierarchy, it is possible e.g. a sensor's geographical details and its readings under the same structure, whereas in a relational database, both are required to be stored in different tables. This is excellent for generating reports where all the data is shown under one heading. More often than not, JSON-based databases use JavaScript as their language for manipulating data, instead of SQL which is used in relational databases. Hence the new jargon called NoSQL has been coined to describe hierarchical databases. Although number crunching is possible in hierarchical databases, relational databases with their clean tabular structure is more suited to such operations. It is not uncommon for solutions to use both relational and hierarchical at the same time. The relational databases are used to record data and crunch the numbers and later moved over to hierarchical databases for reports. The most popular hierarchy database used is MongoDB, since it was the first solution to attack the problem of hierarchy. Nowadays we have other solutions like Cassandra and CouchBase.

Linked Databases or Graphs

When relations between objects play a key role, e.g. for a social network which has relations among persons or for objects such as art or history, where one data is related to another and a researcher could go on to find related or similar pieces, neither table based data, nor hierarchy data fit the bill. One needs to store data in a form where a number of relations are traversed to go from point A to point B. E.g. in a logistics where a shipment must be delivered from Cape Town to Kyoto, one must be able to store and traverse data which represents something like truck from Cape Town to Johannesburg, flight from Johannesburg to Dubai, flight from Dubai to Tokyo and railway freight from Tokyo to Kyoto. Such structures are fulfilled by graph theory of mathematics and special database solutions exist for such problems. Graphs are represented by vertices and edges to inter-connect. Of particular advantage is the ability to assign costs or strengths (also called weights) to the connection between two points which helps in finding the most optimised solution for a given path.

Data represented as a linked database or a graph. The points (called vertices) are the individual objects such as cities, and the connections (also called edges) are the relations between objects, such as an available transport.

The most common database used for graph databases is the Google offering named Cayley. Other solutions such as FreeBase are also available.

Conclusion

This post has dissected the various options available for gathering and storing data as permanent storage records in the cloud. In the next post, we shall see what technologies exist to extract data from the databases once it grows into a monstrous size such as billions or trillions of records.

What makes up the IoT: The digital chatter

Hello Internet. In the last post, we saw some common types of devices on the field, as well as what methods are used to convert them from standalone units to connected devices. Today we will see how the devices actually communicate with each other and how they shoot off their data to the big bad cloud.

Who speaks to who

IoT is a complete ecosystem of devices collaborating with each other to make things smarter. It is important to classify the different types of sources and targets of these collaborations. There are namely the following directions of communication.
Device to device: This involves devices on the field communicating with each other. Typically this happens when one of the devices senses something and it must make sure that all the devices in the field must know about it. Such an intercommunication is called meshing. A network where devices on the field are capable of communicating with each other is called a 'mesh network'.

A typical mesh network
Image courtesy: indigosystems.net.nz

Device to cloud: This is the most frequent communication. Once devices gather info, they must send the info over to the cloud for further analysis. Most of the time, the analysed data never needs to be sent back to the device and is just crunced on the server to show insights to a data analyst. But in some systems, the server needs to process device-sent data immediately and react back to the device. E.g. route calculation on a map to determine the next turn. The system needs to be extremely reactive when the driver takes a wrong turn. Typically, routes are recalculated to keep the driver driving in the same direction instead of making him take a U-turn.
Cloud to device: There are times when a user would like to send instructions to the field to behave a certain way. The same way, some conditions can trigger automated changes to the field. E.g. When a user leaves his office, the heater at his home should be turned on. For these situations, a message is sent over from the cloud towards the field. The field then reacts to such messages and aligns its devices to perform the desired action.

Methods of communication used

Device to device

There are three widespread communication media used for device to device communication: Bluetooth, Zigbee and WiFi. The first two protocols have fixed instructions and fixed formats for the data that can be sent between devices. These are documented in their respective standards. WiFi sets the standards for the hardware, but leaves the data format free to be chosen and hence is preferred over the other two media for a much more generic system. WiFi uses the same communication standards as used by the Internet itself. So by using WiFi, we can think of the devices in the field as a small local network like the one found in your office -- except that these devices operate without much human intervention. In case of WiFi, then methods of communication used are similar to the ones used for Cloud-to-Device, which we will see below.

Cloud to Device

Since the cloud to device communication is the Internet, the standard message passing communications which are used on the Internet are also applicable here. Lets look at the most popular ones.
Chat: Chat protocols are extremely popular and time-tested and can be easily ported over to the IoT scene. Think of the cloud and the devices being on each other's friends list and chatting away all day long. The most popular chat protocol used here is XMPP, also called Jabber, which is the go-to protocol used by popular services like the erstwhile Gtalk (Hangouts has dropped support for Jabber) and Facebook chat. Not only does this protocol offer chat support, but also presence support, i.e. whether a device is offline or online (similar to users). This can be used to determine if a field device has gone down and needs repairs.
Publish / Subscribe: Publish / subscribe is a popular method used by news channels to push the latest news to subscribers. If we use popular news apps on our phones / tablets, we will occasionally see notifications for new news articles. This is publish / subscribe or pub-sub in action. Here the participants subscribe to an entire channel or certain topics and receive notifications when something new is ready. The model can be used on IoT as well, wherein devices can subscribe to new instructions from the cloud and instructions can be sent over. The most popular protocol used here is MQTT (Messaging Queue Telemetry Transport).

Device to Cloud

The cloud to device methods are symmetric methods and can be used for device-to-cloud communications too. But there are a lot of devices in the market which do not need a two way communication and they only need to send data one way (ie. device to cloud), while only receiving an acknowledgement response from the other end. E.g. a weather sensor does not need to pick instructions from the cloud and only needs to send over data. For such purposes, using a two way protocol will be wasteful. So a simple HTTP communication will suffice. HTTP communication is easy to build since it is the most popular protocol on the Internet and the most widely supported. It is easy to find services which offer HTTP off the shelf without installing any custom software modules and drop-dead simple to find engineers to write web services.

Conclusion

We have looked at how the field and the cloud communicate with each other. In the next post, we will see how the cloud gathers the data and what technology is available to store, number crunch and present meaningful and insightful information to the user.

What makes up the Internet of Things: The Field

Hello Internet, I am starting a series of posts from today which look at the various components that make up the Internet of Things. This post has been titled as 'The Field', which basically is the place where the action is taking place. The Ground zero. E.g. If we are making a smart jogging app, then the 'field' is the circuit inside your running shoe or your smartphone, both of which will capture your jogging data.
We are going to look at the various options available and the solutions currently used en masse by the big players for gathering the field data.

Field components

So what makes up the field and what are the roles of the components in the field?

• For an application which requires studying and observing the surroundings WITHOUT user interaction, then field components have attached sensors which read various parameters from the surroundings. E.g. a smart watch has a heartbeat sensor to record heart rate when a user is jogging. A weather reading node has a thermal sensor, humidity sensor and a wind speed sensor.
• For an application WITH user interaction, the field component will have the necessary input units such as a key touchpad, a card swipe slot, a touchscreen, knobs & switches.

Typically both the sensed data and user input data are sent from the device to a control centre or a central data repository for further processing to get an insight into the user behaviour or to learn from the environment surrounding the field.

• Most of the IoT systems are made to be remotely controlled by a user who is not present on the field. E.g. spaceships, CCTV cameras, which are quite common, but now increasingly trendy household appliances such as remote controlled washing machines or microwave ovens.
  Such devices need to be connected to the Internet and need to respond to remote commands by activating their action units. E.g. a sprinkler system will open up its valves, a remote washing machine will start its washing cycle, etc.

Connection hardware used on field components

Most IoT field components have multiple slots for various connections. The most commonly used ones are:
Ethernet / WiFi: These are used to connect with the Internet.

Ethernet port
Image coutesy: terrywhite.com

Device with WiFi antenna
Image courtesy: taoglass.com

Bluetooth: If the device itself does not have Internet connectivity options, then Blutooth is on hand to connect the device to a nearby phone and depend on it for Internet connectivity.

Device with bluetooth
Image courtesy: amazon.com

Power cable / battery: These are used to power up the device.

USB: This is generally used to connect add-on devices or to connect this device to a personal computer.
GPIO: While many standards exist for connecting different types of sensors, input & output units to IoT circuits, GPIO (General Purpose Input / Output) is the most common standard used in IoT for fitting various add-on devices to a base device. The main reason is the form factor, the low power consumption and the fact that a GPIO pin can be configured to do both input & output, according to need.

Common Solutions

Now that we have seen the roles and the connection options used in field components, let us look at the common solutions that are used a field components
Arduino
No other company has captured the imagination of budding engineers than the Arduino. What started itself as a company which wanted to make DIY electronics easier for youngsters was catapulted by mass adoring adoption into the most revered company providing automated electronic solutions for every problem imaginable. Due to the sheer number of add-ons sensors available, from humidity sensors to friction sensors to action cameras, the Arduino microcontroller board is the #1 choice for sensor based IoT solutions. However the Arduino is a microcontroller and not a full fledged processor, since it is made for ultra low power consumption on batteries. This means that one Arduino board can do only one job well, e.g. one cannot connect two or three different types of sensors and expect the Arduino to work well. Secondly, the Arduino has its own operating system and its own programming language and one has to go through a learning curve to start working on both.
Raspberry Pi
The UK based Pi started as a solution for education institutions to teach children how to write computer programs. With the price equivalent US $35 per board, the Pi got record breaking adoptions. Add to this the fact that the circuit of the Pi was published freely on the Internet, companies and individuals started circuit-printing their own Pi's and made sure that the demand never overwhelmed the supply. In one year, the Pi entered mainstream IoT circuits for workshops, exhibitions and industries. Unlike Arduino, the Pi is based on Linux and Python, both of which are heavily used in the industry and hence the learning curve is practically nil for those working in those fields. By embracing Linux, the Pi adopted the IoT industry's #1 operation system and became a hit overnight. The processing power is much higher than an Arduino and there are connection options such as ethernet, USB and GPIO. But the Pi's biggest hit to date has been that it's makers built a hardware bridge which allows it to speak to an Arduino. Instead of competing, the Pi suddenly tripled the possibilities of what can be built for IoT. With the number of sensors available for an Arduino and the higher processing power of Pi, suddenly the IoT solution designers could use the two in combination to build seamless combinations of internet connection high processing sensors.
BeagleBone
Texas Instruments's Sitara processor forms the base of the US based equivalent of the Pi. Started with similar educational purposes in mind, the excellent engineering from TI finally made it a strong case for BeagleBone to be adopted by the IoT bigwigs. At $45 per board, the BeagleBone packs as much processing as a 1990s Pentium processor, well capable of running a GUI based operating system. Keeping this in mind, TI has been actively working on and releasing Android's progressive versions for the BeagleBone, immediately making Android developers feel at ease with development on the BeagleBone. Attaching a touchscreen to the GPIO pins on a BeagleBone makes it a completely ready Android system and one can draw as much functionality from it as from a typical Android phone. Like the Pi, the BeagleBone too can perform good multiprocessing and can in fact act as a mini server if used efficiently. Add-ons for the BeagleBone are currently few and are quite expensive and it is not yet possible to talk to other platforms such as the Arduino.
So that wraps up our discussion on what components are used on the field. In the next post, we will discuss other topics related to the field such as the type of data used and the strategies for data transfer and how remote instructions are handled.

Openness and Standards: The Lingua Franca

Why Openness and Standards

As thousands and millions of smart devices enter our homes, pockets & lives today, there is one thing that we desperately want them to do..... and do soon. Talk to each other. Devices have improved our lives in may ways & yet not many of them can seamlessly talk to each other. Our clock cannot talk to our washing machine and tell it to start the washing cycle at a predefined time. Nor can our fridge talk to our cell phone to remind us to get milk on the way home.
A lot of companies such as Nest, Samsung, Sony, Apple and scores of others have come up with solutions to make their devices talk among themselves and make things seamless. But alas, all these solutions work only within the products made by the same company. Call it ego, call it business strategy or call it the complete failure by the big names to have the humble magnanimity to admit that they cannot roll out a complete set of products which are superior to the offerings by other companies, but the consumer is currently stuck with a situation where he / she cannot pick the best device from each company and have them talk to each other. The vendor tie-up is generally due to various reasons, such as a proprietary cable (e.g. iPhone's data cable / charger), a proprietary data transfer method (e.g. Samsung Kies, iTunes) and other such reasons which locking solution to products by the same company.

Nest Thermostat syncs with your phone, but uses its own app, which needs Android or iOS. One cannot use a web browser, chat or SMS to control things yet. Image courtesy: slashgear.com

Samsung Gear is the most feature rich smart watch of the day, but it connects and syncs with Samsung Galaxy 4 or higher ONLY.
Image courtesy: sammobile.com

One of the biggest advantages to be fulfilled by IoT is the principle of 'one-touch setup & forget'. The devices should work so seamlessly that the user is not even aware of their existence. Any device can connect & share data with any other device or with a central unit. At the end of the day, IoT's main purpose is to automate our lives and drain the mundane away from the human user. The buzzword for IoT is exactly summarized in one word -- compatibility. Exactly like what lovers seek in each other. Users seek compatibility among devices. This word has long been replaced by another jargon -- Interoperability. A smart device doing its assigned job well is so 2000s. In 2015, we need devices that function well and also interoperate well with other devices that need their services. A connected world needs connected devices, but with no one having to yank things together hard to establish the connection. So you have designed a smart washing machine which uses cutting edge technology to minimize detergent usage and use water efficiently? Not enough. It should be able to talk to your smart phone, your laptop and possibly even your smart watch, wherever you are and whatever you are doing, telling you about the progress of washing or alert you if it's detergent bay is empty or if it is not able to draw water. It should be able to alert you via mass-used methods such as Gtalk or Whatsapp or even SMS.

Standardisation as panacea

How can such a seamless level of interoperability be achieved? One simple magical word: Follow the standards. Big companies and consortiums have long existed to draw up standards for how devices should be designed and how they should communicate with each other. But most of the standards are in their infancy and not getting the necessary push needed to kick them into mainstream, since a lot of companies are happy to hit the market first with their own proprietary technologies. This is counter-productive for them, since the majority of users end up rejecting the product for its lack of interoperability.
Standards are the lingua franca which allow different devices to talk a common language. How are standards made in the first place? Multiple companies and experts form a consortium to identify if a set of functionalities have common threads. Case in point: HTML. Companies like Netscape, Microsoft and Mozilla came together to decide on what are the requirements for a typical web page and how browsers should show them. So over the years, we now have a standard language called HTML which defines how a web page looks and how each component should be represented in a simple English-like language. Anyone who wants to make his / her own web browser now knows exactly what to do when HTML is fed to his / her browser, since each and everything is standardized by the governing consortium.

Under the World Wide Web Consortium (W3C), HTML and CSS as standards have set the trail blazing for what constitutes a web page, how things are laid out, how they should look and what can be in a typical web page and how they should be rendered by a web browser that claims to support HTML and CSS. As one can see, Apple's Safari, Opera, Google's Chrome, Mozilla's Firefox and Microsoft's Internet Explorer are the major browsers which have taken a stand to adhere to W3C's standards. HTML is currently standardized as version 5 and CSS as version 3. Both these versions are major rehauls over their predecessors, including support for streaming video & audio, 3D rendering and jerk free graphics.
Image Courtesy: ronakweb.com

Openness as a way to speed up standards adoption

How can standards be evolved? By making participation open and inviting as many parties to use them as possible. Over the years, new additions can be suggested when proof of concepts are made to work and they are incorporated in the standards. After 23 years, HTML has hit version 5. From a simple language which was capable of only defining behaviour for text, hyperlinks & static images, HTML today has inbuilt components for laying out 3D animations and streaming videos. This was possible because web and HTML were released to the entire world to develop, hone & mature. More such examples include hardware standards such as USB, software standards such as ODBC for database manipulation, communication standard protocols such as SMTP for email and file format standards such as PNG for images.

Open standards in IoT

Standards have recently been suggested for the IoT field as well, based on observations of common behaviour by many of the prevailing devices. There are some functionalities which are common to all IoT components regardless of their goal, e.g. connection with other devices, sending & receiving data, analyzing, aggregating, etc. Open standards already exist for each of these functionalities and the companies which implement these will offer maximum value with their devices by sticking to those standards.
Open standards can be classified at 4 layers: the hardware, the software, the communication and the container formats (file formats).
Lets look at each layer and which open solutions exist at each layer.

Open hardware standards

Things become easy when the hardware itself is standardized. This governs what the circuit should contain, how many wires are to be used, which wire is to be used for what purpose, how much amount of current should pass through each wire, etc. The most rampantly used hardware standards today are USB, Ethernet, WiFi and bluetooth. Most devices will be immediately within the standards as long as they follow 2 or more of the above mentioned technologies. However another wire-free hardware standard called Zigbee is fairly popular in the home automation field. The above standards stand testimony to the fact that an openly published standard is adopted my millions of participants and a whole ecosystem is created as a result. E.g. we get everything from staple things like USB keyboards to specialized things like USB connected cooling fans. Same way, ethernet is the backbone hardware for technologies dominating the internet today. Everything from surfing a web page to watching a stream video are built upon current passing through ethernet cables.

Bluetooth has made wireless communication among devices within short distances a snap
Image courtesy: iconarchive.com

Ethernet has lived up to its name by having its presence almost everywhere and on everything synonymous with Internet
Image courtesy: sevacall.com

USB has spread in such a versatile manner that it is now the most expected and the most commonly offered wired connection solution between devices. Everything from storage devices to portable charges to portable cooling fans are driven by USB.
Image courtesy: travelinlibrarian.info

Some companies have gone to the length of publishing their entire circuitry for use by the public. Anyone can download the specification and make the circuitry on his / her own. Publishing the entire specification also allows modifications to the circuit or making add-on devices with a better understanding of the original circuit and hence higher interoperability. Two companies who dominate the field of open hardware at the moment are the US-based BeagleBoard and the UK-based Raspberry Pi. While the Pi consumes lower power and has more variety of add-ons dedicated to it, the BeagleBoard is much more versatile and powerful and a single board can support multiple add-ons. Over the years, the Pi and BeagleBoard are projected to be the base boards powering millions of IoT projects around the world.

BeagleBone Black (above) and Raspberry Pi (below) started as hobby and education devices, but made a strong case for startups to latch onto them for their IoT ideas mainly due to their Open Hardware approach wherein they published their entire circuitry details for anyone to reproduce, modify & adapt. Besides all the connectivity is done using standard hardware approaches such as ethernet and USB.
Image courtesy: linuxgizmo.com and wikipedia.org

Open software standards

The 1980s and 1990s saw companies like Microsoft and Adobe dominate the desktop software scenario. Their software was proprietary, sold on CD/DVD and installed as per the setup instructions, working only on the operation system and the hardware supported. From early 2000s, a new trend emerged. With handheld devices being used and a single person possessing multiple devices like a laptop, a phone and a tablet, software consistently needed to be written for multiple types of hardware platforms with different limits of resources such as memory & storage space. In fact, user's data was moved off hard disks and straight to the internet into the cloud. The trend started with emails, where desktop email software such as Outlook was replaced by the browser. Now we have entire Office suites and photo editors running entirely in browsers, be it the desktop, mobile or tablet... and even television and automotive dashboards!
Such a mass transformation needs base software which can be freely moulded to fit absolutely any type of hardware available. Also, when new types of hardware emerges, this software must lend itself to free transformation. Such software cannot be managed by ONE single company, nor by licenses. ENTER LINUX! This is the operating system of choice for 99% of startups and IoT devices. Android is also a form of Linux and its success doesn't need any introduction. But even though we see so many varieties of Linux, they all follow a standard in their core functionality. The standards are documented as POSIX standards and all Linux variants are required to follow them, so that the developers don't get any nasty surprises while making the necessary software. Similarly, software components such as web browsers, web servers, email clients & servers, chat, etc all typically build from existing open source software which adhere to standards.

Android (top) and Apache (middle) have built on the openness and success of Linux (bottom) to offer software solutions which are used in more than 90% of the IoT and smart devices industry.
Image courtesy: davy.preuveneers.com, brighthub.com and wikipedia.org 

Open communication standards

If one were to build his / her own satellite communication device or SIM card, chances are that he / she will be arrested for his / her efforts. This is because satellite communication methods are guarded and licensed. Only authorized personnel can make hardware or tune into those systems. This is an example of closed communication. However for the purpose of IoT, we will stick to the most open and standard communication methods out there. Some of those were discussed in the hardware section, such as bluetooth, WiFi, Ethernet and Zigbee. These communication methods are license free and anyone can make devices and software which use these media. Similarly, protocols which build on top of these media can be open or closed. E.g. HTTP (web surfing), SMTP (email), Jabber & XMPP (chat) are all standard & open, but some software like Skype and Netflix use their own proprietary transfer methods. Bluetooth has also taken big strides to achieve standard behaviour sets by classifying devices as different profiles such as phone, headset, streaming device, etc. Devices are required to adhere to one of the pre-defined profiles according to the version of Bluetooth that they claim to support.

HTTP for web surfing and Bluetooth Audio for streaming music over bluetooth from a music player to a wireless headphone are two communication and data transfer standards that have emerged from the success of Ethernet and Bluetooth hardware standards.
Images courtesy: novainfosec.com and marmitek.com

Open container formats (file formats)

While some file storage formats are openly documented, some of them are licensed to companies. HTML, JSON, XML, vCard and PNG are some formats which are free for use by anyone, whereas doc, ppt, pdf and mp3 are formats which are licensed by certain companies. One cannot write their own tool to read or write in these formats and must either use the source company provided and licensed tool / algorithm or get a license to write their own tool.

Portable Network Graphics (PNG), JavaScript Object Notation (JSON) and vCard are some of the open data storage and transfer formats which have gained popularity for transferring different types of data between devices and applications. E.g. PNG is used to transfer and store images such as logos, vCard is used to transfer information about contacts & people (including their contact details such as email and phone number) and JSON has become the de facto format for transferring and storing any arbitrary information not covered by any of the standard formats. JSON is also THE standard format used by search engines to spider through a website's data to rank in search results. SEO optimization is all about getting the JSON meta-data correct. The JSON formats needed by search engines to identify different types of objects (e.g. a person, a music track, a tourist destination) is documented in a seperate standard called JSON-LD (JSON Linked Data).
Images courtesy: creativitywindow.com, encoding.com, gettinglegaldone.com

Conclusion

So as one can see, for free communication and data sharing among the participants of IoT, one needs to implement components which follow standards. With more adherence, more interoperability can be achieved.
In the blogs to come, I will discuss which open options are available at each level, so that one can implement a seamless, interoperable ecosystem of IoT devices.