Gathering and recording data from the field

We have seen in the previous blogs how devices on the field shoot out data to the cloud. We have also seen how the cloud can instruct the field to carry out remote instructions. We will now see what one uses on the cloud in order to set up the infrastructure to manage huge reams of field data.

What exactly is the cloud

Jargon speak has introduced a lot of words these days and the word 'cloud' is even more prevalent than 'IoT'. To super-simplify the definition, think of it this way. Instead of setting up our own infrastructure which would need 24x7 electricity, Internet connection, software and hardware maintenance and thus a humungous effort and a support team, we hand off all the headache to a provider who takes care of all these. This provider will make sure that he offers the following features:

• Server machines where our services will be hosted. The biggest feature is that server machines can be added quickly on the fly if the service's loads are taking a big hit during peak hours.
• Basic or complete suite of software which is needed for our services. Most providers will provide plans such as web hosting package or Web/Database combo package. Some of them allow customization to allow us to install our own software. Almost all of them have plans which set up the basics and then offer us a way to access the machines so that we can tweak them as we see fit.
• 24x7 availability with support. While a 100% availability is not guaranteed, most services go for 99 or 99.9% uptime.
• Payment plans which are generally very economical. Most of these services offer pay-as-much-as-you-use model. Almost all of them offer an advance lock-in for a few months or years at much cheaper rates if we commit to hosting our services on them for a long term. E.g. if the service costs 3 cents per hour of uptime, a service provide may agree at 2 cents per hour if we commit to hosting with them for the next 6 months.
• Dashboards to control, monitor and oversee the performance and tweak settings and add/remove servers on the fly.

The most popular cloud services at the moment are Amazon & Rackspace. Both of them offer Windows and Linux servers. Google App Engine and Google Cloud Computing have made huge inroads in the last 2 years with their super high performing and highly standards compliant offerings, but their solutions are Linux-only and hence there is no support for .NET based applications. Microsoft on the other hand offers the Windows-only Azure.

Image Courtesy: rackwareinc.com

Image Courtesy: digitalmunition.me

Image courtesy: cloud.google.com

Image courtesy: cloudtimes.org

This is by no means a complete list and there are many local and regional players in the game, such as Netmagic here in India. In Europe and USA, almost every Internet Service Provider also doubles up as a cloud provider.

What comes across from the field to the cloud

At the moment, there is no standardisation in the protocol or format with which data can be sent over from the field to the cloud. HTTP is the de facto protocol of choice since it has been around for far longer than the other protocols. Tried and tested, it is easy to get the man power to churn out a HTTP based data transfer service, since the World Wide Web uses the same protocol. However it is not uncommon to see other solutions such as Jabber (the most common chat protocol) or MQTT (the latest IoT specific data transfer protocol).

As far as the format of data goes, different vendors use different ways to represent the data. JSON is the most common format, followed by XML. However, many solutions use their own proprietary format, specially tailored for their IoT system and not interoperable with any of the others.

JSON and XML are particularly popular because of the ability to represent one piece of data inside another and thus achieve a hierarchy and an order of structure, which is how objects exist in the real world. However JSON has recently gained more popularity over XML since JSON can represent the same data in less number of bytes and is hence seen as network friendly for slow GPRS based networks. Some companies stick to XML though, since data represented in XML can be validated before being accepted on the server using a standard technique called DTD, without writing a line of software. No such standard standard validation technique exists for JSON and companies much implement their own validation mechanism by writing their own code / algorithm.

Typical JSON data showing employee records

Typical XML data showing book records

While JSON is receiving rapid adoption, there is no common JSON structure that is standardized across two vendors, thus defeating the purpose of JSON's interoperability across systems. E.g. In the employee structure above, one could split name as 'firstName' and 'lastName' by another solution. A system which is programmed to read 'name' would fail to recognise the split names.
To solve this, IoT has been increasingly tapping into a new standard called schema.org. This new standard has defined the JSON structure for many common type of 'things' such as an employee. Conforming to these pre-standardised structures will allow systems from various vendors to interoperate and synergistically work together.

Schema.org JSON structure standardised for a store. Any software which needs to represent a store should ideally follow this structure and that sofware will find it easy to work with stores all over the Internet.

How the cloud stores data

The data that comes from the field is fleeting and needs to be recorded somewhere for further analysis. There are several methods available to record and store the data. The obvious choice to do so is in databases. But there are so many types and brands of databases available that must be evaluated, compared and selected to arrive upon the best solution.

Broadly there are three main types of databases available these days: Traditional relational databases which have been around since the Darwinian days of digitally recorded data, hierarchical databases (typified by the term named 'NoSQL' these days) and linked databases (also called Graph).

Relational Databases

Avoiding a highly technical definition, let's just say that in relational databases, data is stored in tabular rows and columns, each table representing an entity or a 'thing', such as an employee. One table can link to another related table using a method called 'joining'. E.g. If there is one table storing all employee details such as his personal and contact details and another which stores his/her income and tax details, then for making a tax statement which shows the employee name and his/her tax liabilities, the two tables must be joined to fetch the info and make the report. Joining is computationally expensive and has been known to slow down data fetching operations. While relational databases are excellent for crunching transactional records into summaries very quickly, fetching data from multiple tables for the purpose of reporting is a massive bottleneck.

The most popular database solutions in the market currently are Oracle and MySQL. Microsoft based systems use Microsoft SQL server.

Hierarchical Databases

Borrowing from the concept of JSON described above, hierarchical databases use JSON structure to nest different types of data into each other, thus eliminating joins. Using hierarchy, it is possible e.g. a sensor's geographical details and its readings under the same structure, whereas in a relational database, both are required to be stored in different tables. This is excellent for generating reports where all the data is shown under one heading. More often than not, JSON-based databases use JavaScript as their language for manipulating data, instead of SQL which is used in relational databases. Hence the new jargon called NoSQL has been coined to describe hierarchical databases. Although number crunching is possible in hierarchical databases, relational databases with their clean tabular structure is more suited to such operations. It is not uncommon for solutions to use both relational and hierarchical at the same time. The relational databases are used to record data and crunch the numbers and later moved over to hierarchical databases for reports. The most popular hierarchy database used is MongoDB, since it was the first solution to attack the problem of hierarchy. Nowadays we have other solutions like Cassandra and CouchBase.

Linked Databases or Graphs

When relations between objects play a key role, e.g. for a social network which has relations among persons or for objects such as art or history, where one data is related to another and a researcher could go on to find related or similar pieces, neither table based data, nor hierarchy data fit the bill. One needs to store data in a form where a number of relations are traversed to go from point A to point B. E.g. in a logistics where a shipment must be delivered from Cape Town to Kyoto, one must be able to store and traverse data which represents something like truck from Cape Town to Johannesburg, flight from Johannesburg to Dubai, flight from Dubai to Tokyo and railway freight from Tokyo to Kyoto. Such structures are fulfilled by graph theory of mathematics and special database solutions exist for such problems. Graphs are represented by vertices and edges to inter-connect. Of particular advantage is the ability to assign costs or strengths (also called weights) to the connection between two points which helps in finding the most optimised solution for a given path.

Data represented as a linked database or a graph. The points (called vertices) are the individual objects such as cities, and the connections (also called edges) are the relations between objects, such as an available transport.

The most common database used for graph databases is the Google offering named Cayley. Other solutions such as FreeBase are also available.

Conclusion

This post has dissected the various options available for gathering and storing data as permanent storage records in the cloud. In the next post, we shall see what technologies exist to extract data from the databases once it grows into a monstrous size such as billions or trillions of records.

What makes up the IoT: The digital chatter

Hello Internet. In the last post, we saw some common types of devices on the field, as well as what methods are used to convert them from standalone units to connected devices. Today we will see how the devices actually communicate with each other and how they shoot off their data to the big bad cloud.

Who speaks to who

IoT is a complete ecosystem of devices collaborating with each other to make things smarter. It is important to classify the different types of sources and targets of these collaborations. There are namely the following directions of communication.
Device to device: This involves devices on the field communicating with each other. Typically this happens when one of the devices senses something and it must make sure that all the devices in the field must know about it. Such an intercommunication is called meshing. A network where devices on the field are capable of communicating with each other is called a 'mesh network'.

A typical mesh network
Image courtesy: indigosystems.net.nz

Device to cloud: This is the most frequent communication. Once devices gather info, they must send the info over to the cloud for further analysis. Most of the time, the analysed data never needs to be sent back to the device and is just crunced on the server to show insights to a data analyst. But in some systems, the server needs to process device-sent data immediately and react back to the device. E.g. route calculation on a map to determine the next turn. The system needs to be extremely reactive when the driver takes a wrong turn. Typically, routes are recalculated to keep the driver driving in the same direction instead of making him take a U-turn.
Cloud to device: There are times when a user would like to send instructions to the field to behave a certain way. The same way, some conditions can trigger automated changes to the field. E.g. When a user leaves his office, the heater at his home should be turned on. For these situations, a message is sent over from the cloud towards the field. The field then reacts to such messages and aligns its devices to perform the desired action.

Methods of communication used

Device to device

There are three widespread communication media used for device to device communication: Bluetooth, Zigbee and WiFi. The first two protocols have fixed instructions and fixed formats for the data that can be sent between devices. These are documented in their respective standards. WiFi sets the standards for the hardware, but leaves the data format free to be chosen and hence is preferred over the other two media for a much more generic system. WiFi uses the same communication standards as used by the Internet itself. So by using WiFi, we can think of the devices in the field as a small local network like the one found in your office -- except that these devices operate without much human intervention. In case of WiFi, then methods of communication used are similar to the ones used for Cloud-to-Device, which we will see below.

Cloud to Device

Since the cloud to device communication is the Internet, the standard message passing communications which are used on the Internet are also applicable here. Lets look at the most popular ones.
Chat: Chat protocols are extremely popular and time-tested and can be easily ported over to the IoT scene. Think of the cloud and the devices being on each other's friends list and chatting away all day long. The most popular chat protocol used here is XMPP, also called Jabber, which is the go-to protocol used by popular services like the erstwhile Gtalk (Hangouts has dropped support for Jabber) and Facebook chat. Not only does this protocol offer chat support, but also presence support, i.e. whether a device is offline or online (similar to users). This can be used to determine if a field device has gone down and needs repairs.
Publish / Subscribe: Publish / subscribe is a popular method used by news channels to push the latest news to subscribers. If we use popular news apps on our phones / tablets, we will occasionally see notifications for new news articles. This is publish / subscribe or pub-sub in action. Here the participants subscribe to an entire channel or certain topics and receive notifications when something new is ready. The model can be used on IoT as well, wherein devices can subscribe to new instructions from the cloud and instructions can be sent over. The most popular protocol used here is MQTT (Messaging Queue Telemetry Transport).

Device to Cloud

The cloud to device methods are symmetric methods and can be used for device-to-cloud communications too. But there are a lot of devices in the market which do not need a two way communication and they only need to send data one way (ie. device to cloud), while only receiving an acknowledgement response from the other end. E.g. a weather sensor does not need to pick instructions from the cloud and only needs to send over data. For such purposes, using a two way protocol will be wasteful. So a simple HTTP communication will suffice. HTTP communication is easy to build since it is the most popular protocol on the Internet and the most widely supported. It is easy to find services which offer HTTP off the shelf without installing any custom software modules and drop-dead simple to find engineers to write web services.

Conclusion

We have looked at how the field and the cloud communicate with each other. In the next post, we will see how the cloud gathers the data and what technology is available to store, number crunch and present meaningful and insightful information to the user.

What makes up the Internet of Things: The Field

Hello Internet, I am starting a series of posts from today which look at the various components that make up the Internet of Things. This post has been titled as 'The Field', which basically is the place where the action is taking place. The Ground zero. E.g. If we are making a smart jogging app, then the 'field' is the circuit inside your running shoe or your smartphone, both of which will capture your jogging data.
We are going to look at the various options available and the solutions currently used en masse by the big players for gathering the field data.

Field components

So what makes up the field and what are the roles of the components in the field?

• For an application which requires studying and observing the surroundings WITHOUT user interaction, then field components have attached sensors which read various parameters from the surroundings. E.g. a smart watch has a heartbeat sensor to record heart rate when a user is jogging. A weather reading node has a thermal sensor, humidity sensor and a wind speed sensor.
• For an application WITH user interaction, the field component will have the necessary input units such as a key touchpad, a card swipe slot, a touchscreen, knobs & switches.

Typically both the sensed data and user input data are sent from the device to a control centre or a central data repository for further processing to get an insight into the user behaviour or to learn from the environment surrounding the field.

• Most of the IoT systems are made to be remotely controlled by a user who is not present on the field. E.g. spaceships, CCTV cameras, which are quite common, but now increasingly trendy household appliances such as remote controlled washing machines or microwave ovens.
  Such devices need to be connected to the Internet and need to respond to remote commands by activating their action units. E.g. a sprinkler system will open up its valves, a remote washing machine will start its washing cycle, etc.

Connection hardware used on field components

Most IoT field components have multiple slots for various connections. The most commonly used ones are:
Ethernet / WiFi: These are used to connect with the Internet.

Ethernet port
Image coutesy: terrywhite.com

Device with WiFi antenna
Image courtesy: taoglass.com

Bluetooth: If the device itself does not have Internet connectivity options, then Blutooth is on hand to connect the device to a nearby phone and depend on it for Internet connectivity.

Device with bluetooth
Image courtesy: amazon.com

Power cable / battery: These are used to power up the device.

USB: This is generally used to connect add-on devices or to connect this device to a personal computer.
GPIO: While many standards exist for connecting different types of sensors, input & output units to IoT circuits, GPIO (General Purpose Input / Output) is the most common standard used in IoT for fitting various add-on devices to a base device. The main reason is the form factor, the low power consumption and the fact that a GPIO pin can be configured to do both input & output, according to need.

Common Solutions

Now that we have seen the roles and the connection options used in field components, let us look at the common solutions that are used a field components
Arduino
No other company has captured the imagination of budding engineers than the Arduino. What started itself as a company which wanted to make DIY electronics easier for youngsters was catapulted by mass adoring adoption into the most revered company providing automated electronic solutions for every problem imaginable. Due to the sheer number of add-ons sensors available, from humidity sensors to friction sensors to action cameras, the Arduino microcontroller board is the #1 choice for sensor based IoT solutions. However the Arduino is a microcontroller and not a full fledged processor, since it is made for ultra low power consumption on batteries. This means that one Arduino board can do only one job well, e.g. one cannot connect two or three different types of sensors and expect the Arduino to work well. Secondly, the Arduino has its own operating system and its own programming language and one has to go through a learning curve to start working on both.
Raspberry Pi
The UK based Pi started as a solution for education institutions to teach children how to write computer programs. With the price equivalent US $35 per board, the Pi got record breaking adoptions. Add to this the fact that the circuit of the Pi was published freely on the Internet, companies and individuals started circuit-printing their own Pi's and made sure that the demand never overwhelmed the supply. In one year, the Pi entered mainstream IoT circuits for workshops, exhibitions and industries. Unlike Arduino, the Pi is based on Linux and Python, both of which are heavily used in the industry and hence the learning curve is practically nil for those working in those fields. By embracing Linux, the Pi adopted the IoT industry's #1 operation system and became a hit overnight. The processing power is much higher than an Arduino and there are connection options such as ethernet, USB and GPIO. But the Pi's biggest hit to date has been that it's makers built a hardware bridge which allows it to speak to an Arduino. Instead of competing, the Pi suddenly tripled the possibilities of what can be built for IoT. With the number of sensors available for an Arduino and the higher processing power of Pi, suddenly the IoT solution designers could use the two in combination to build seamless combinations of internet connection high processing sensors.
BeagleBone
Texas Instruments's Sitara processor forms the base of the US based equivalent of the Pi. Started with similar educational purposes in mind, the excellent engineering from TI finally made it a strong case for BeagleBone to be adopted by the IoT bigwigs. At $45 per board, the BeagleBone packs as much processing as a 1990s Pentium processor, well capable of running a GUI based operating system. Keeping this in mind, TI has been actively working on and releasing Android's progressive versions for the BeagleBone, immediately making Android developers feel at ease with development on the BeagleBone. Attaching a touchscreen to the GPIO pins on a BeagleBone makes it a completely ready Android system and one can draw as much functionality from it as from a typical Android phone. Like the Pi, the BeagleBone too can perform good multiprocessing and can in fact act as a mini server if used efficiently. Add-ons for the BeagleBone are currently few and are quite expensive and it is not yet possible to talk to other platforms such as the Arduino.
So that wraps up our discussion on what components are used on the field. In the next post, we will discuss other topics related to the field such as the type of data used and the strategies for data transfer and how remote instructions are handled.

Openness and Standards: The Lingua Franca

Why Openness and Standards

As thousands and millions of smart devices enter our homes, pockets & lives today, there is one thing that we desperately want them to do..... and do soon. Talk to each other. Devices have improved our lives in may ways & yet not many of them can seamlessly talk to each other. Our clock cannot talk to our washing machine and tell it to start the washing cycle at a predefined time. Nor can our fridge talk to our cell phone to remind us to get milk on the way home.
A lot of companies such as Nest, Samsung, Sony, Apple and scores of others have come up with solutions to make their devices talk among themselves and make things seamless. But alas, all these solutions work only within the products made by the same company. Call it ego, call it business strategy or call it the complete failure by the big names to have the humble magnanimity to admit that they cannot roll out a complete set of products which are superior to the offerings by other companies, but the consumer is currently stuck with a situation where he / she cannot pick the best device from each company and have them talk to each other. The vendor tie-up is generally due to various reasons, such as a proprietary cable (e.g. iPhone's data cable / charger), a proprietary data transfer method (e.g. Samsung Kies, iTunes) and other such reasons which locking solution to products by the same company.

Nest Thermostat syncs with your phone, but uses its own app, which needs Android or iOS. One cannot use a web browser, chat or SMS to control things yet. Image courtesy: slashgear.com

Samsung Gear is the most feature rich smart watch of the day, but it connects and syncs with Samsung Galaxy 4 or higher ONLY.
Image courtesy: sammobile.com

One of the biggest advantages to be fulfilled by IoT is the principle of 'one-touch setup & forget'. The devices should work so seamlessly that the user is not even aware of their existence. Any device can connect & share data with any other device or with a central unit. At the end of the day, IoT's main purpose is to automate our lives and drain the mundane away from the human user. The buzzword for IoT is exactly summarized in one word -- compatibility. Exactly like what lovers seek in each other. Users seek compatibility among devices. This word has long been replaced by another jargon -- Interoperability. A smart device doing its assigned job well is so 2000s. In 2015, we need devices that function well and also interoperate well with other devices that need their services. A connected world needs connected devices, but with no one having to yank things together hard to establish the connection. So you have designed a smart washing machine which uses cutting edge technology to minimize detergent usage and use water efficiently? Not enough. It should be able to talk to your smart phone, your laptop and possibly even your smart watch, wherever you are and whatever you are doing, telling you about the progress of washing or alert you if it's detergent bay is empty or if it is not able to draw water. It should be able to alert you via mass-used methods such as Gtalk or Whatsapp or even SMS.

Standardisation as panacea

How can such a seamless level of interoperability be achieved? One simple magical word: Follow the standards. Big companies and consortiums have long existed to draw up standards for how devices should be designed and how they should communicate with each other. But most of the standards are in their infancy and not getting the necessary push needed to kick them into mainstream, since a lot of companies are happy to hit the market first with their own proprietary technologies. This is counter-productive for them, since the majority of users end up rejecting the product for its lack of interoperability.
Standards are the lingua franca which allow different devices to talk a common language. How are standards made in the first place? Multiple companies and experts form a consortium to identify if a set of functionalities have common threads. Case in point: HTML. Companies like Netscape, Microsoft and Mozilla came together to decide on what are the requirements for a typical web page and how browsers should show them. So over the years, we now have a standard language called HTML which defines how a web page looks and how each component should be represented in a simple English-like language. Anyone who wants to make his / her own web browser now knows exactly what to do when HTML is fed to his / her browser, since each and everything is standardized by the governing consortium.

Under the World Wide Web Consortium (W3C), HTML and CSS as standards have set the trail blazing for what constitutes a web page, how things are laid out, how they should look and what can be in a typical web page and how they should be rendered by a web browser that claims to support HTML and CSS. As one can see, Apple's Safari, Opera, Google's Chrome, Mozilla's Firefox and Microsoft's Internet Explorer are the major browsers which have taken a stand to adhere to W3C's standards. HTML is currently standardized as version 5 and CSS as version 3. Both these versions are major rehauls over their predecessors, including support for streaming video & audio, 3D rendering and jerk free graphics.
Image Courtesy: ronakweb.com

Openness as a way to speed up standards adoption

How can standards be evolved? By making participation open and inviting as many parties to use them as possible. Over the years, new additions can be suggested when proof of concepts are made to work and they are incorporated in the standards. After 23 years, HTML has hit version 5. From a simple language which was capable of only defining behaviour for text, hyperlinks & static images, HTML today has inbuilt components for laying out 3D animations and streaming videos. This was possible because web and HTML were released to the entire world to develop, hone & mature. More such examples include hardware standards such as USB, software standards such as ODBC for database manipulation, communication standard protocols such as SMTP for email and file format standards such as PNG for images.

Open standards in IoT

Standards have recently been suggested for the IoT field as well, based on observations of common behaviour by many of the prevailing devices. There are some functionalities which are common to all IoT components regardless of their goal, e.g. connection with other devices, sending & receiving data, analyzing, aggregating, etc. Open standards already exist for each of these functionalities and the companies which implement these will offer maximum value with their devices by sticking to those standards.
Open standards can be classified at 4 layers: the hardware, the software, the communication and the container formats (file formats).
Lets look at each layer and which open solutions exist at each layer.

Open hardware standards

Things become easy when the hardware itself is standardized. This governs what the circuit should contain, how many wires are to be used, which wire is to be used for what purpose, how much amount of current should pass through each wire, etc. The most rampantly used hardware standards today are USB, Ethernet, WiFi and bluetooth. Most devices will be immediately within the standards as long as they follow 2 or more of the above mentioned technologies. However another wire-free hardware standard called Zigbee is fairly popular in the home automation field. The above standards stand testimony to the fact that an openly published standard is adopted my millions of participants and a whole ecosystem is created as a result. E.g. we get everything from staple things like USB keyboards to specialized things like USB connected cooling fans. Same way, ethernet is the backbone hardware for technologies dominating the internet today. Everything from surfing a web page to watching a stream video are built upon current passing through ethernet cables.

Bluetooth has made wireless communication among devices within short distances a snap
Image courtesy: iconarchive.com

Ethernet has lived up to its name by having its presence almost everywhere and on everything synonymous with Internet
Image courtesy: sevacall.com

USB has spread in such a versatile manner that it is now the most expected and the most commonly offered wired connection solution between devices. Everything from storage devices to portable charges to portable cooling fans are driven by USB.
Image courtesy: travelinlibrarian.info

Some companies have gone to the length of publishing their entire circuitry for use by the public. Anyone can download the specification and make the circuitry on his / her own. Publishing the entire specification also allows modifications to the circuit or making add-on devices with a better understanding of the original circuit and hence higher interoperability. Two companies who dominate the field of open hardware at the moment are the US-based BeagleBoard and the UK-based Raspberry Pi. While the Pi consumes lower power and has more variety of add-ons dedicated to it, the BeagleBoard is much more versatile and powerful and a single board can support multiple add-ons. Over the years, the Pi and BeagleBoard are projected to be the base boards powering millions of IoT projects around the world.

BeagleBone Black (above) and Raspberry Pi (below) started as hobby and education devices, but made a strong case for startups to latch onto them for their IoT ideas mainly due to their Open Hardware approach wherein they published their entire circuitry details for anyone to reproduce, modify & adapt. Besides all the connectivity is done using standard hardware approaches such as ethernet and USB.
Image courtesy: linuxgizmo.com and wikipedia.org

Open software standards

The 1980s and 1990s saw companies like Microsoft and Adobe dominate the desktop software scenario. Their software was proprietary, sold on CD/DVD and installed as per the setup instructions, working only on the operation system and the hardware supported. From early 2000s, a new trend emerged. With handheld devices being used and a single person possessing multiple devices like a laptop, a phone and a tablet, software consistently needed to be written for multiple types of hardware platforms with different limits of resources such as memory & storage space. In fact, user's data was moved off hard disks and straight to the internet into the cloud. The trend started with emails, where desktop email software such as Outlook was replaced by the browser. Now we have entire Office suites and photo editors running entirely in browsers, be it the desktop, mobile or tablet... and even television and automotive dashboards!
Such a mass transformation needs base software which can be freely moulded to fit absolutely any type of hardware available. Also, when new types of hardware emerges, this software must lend itself to free transformation. Such software cannot be managed by ONE single company, nor by licenses. ENTER LINUX! This is the operating system of choice for 99% of startups and IoT devices. Android is also a form of Linux and its success doesn't need any introduction. But even though we see so many varieties of Linux, they all follow a standard in their core functionality. The standards are documented as POSIX standards and all Linux variants are required to follow them, so that the developers don't get any nasty surprises while making the necessary software. Similarly, software components such as web browsers, web servers, email clients & servers, chat, etc all typically build from existing open source software which adhere to standards.

Android (top) and Apache (middle) have built on the openness and success of Linux (bottom) to offer software solutions which are used in more than 90% of the IoT and smart devices industry.
Image courtesy: davy.preuveneers.com, brighthub.com and wikipedia.org 

Open communication standards

If one were to build his / her own satellite communication device or SIM card, chances are that he / she will be arrested for his / her efforts. This is because satellite communication methods are guarded and licensed. Only authorized personnel can make hardware or tune into those systems. This is an example of closed communication. However for the purpose of IoT, we will stick to the most open and standard communication methods out there. Some of those were discussed in the hardware section, such as bluetooth, WiFi, Ethernet and Zigbee. These communication methods are license free and anyone can make devices and software which use these media. Similarly, protocols which build on top of these media can be open or closed. E.g. HTTP (web surfing), SMTP (email), Jabber & XMPP (chat) are all standard & open, but some software like Skype and Netflix use their own proprietary transfer methods. Bluetooth has also taken big strides to achieve standard behaviour sets by classifying devices as different profiles such as phone, headset, streaming device, etc. Devices are required to adhere to one of the pre-defined profiles according to the version of Bluetooth that they claim to support.

HTTP for web surfing and Bluetooth Audio for streaming music over bluetooth from a music player to a wireless headphone are two communication and data transfer standards that have emerged from the success of Ethernet and Bluetooth hardware standards.
Images courtesy: novainfosec.com and marmitek.com

Open container formats (file formats)

While some file storage formats are openly documented, some of them are licensed to companies. HTML, JSON, XML, vCard and PNG are some formats which are free for use by anyone, whereas doc, ppt, pdf and mp3 are formats which are licensed by certain companies. One cannot write their own tool to read or write in these formats and must either use the source company provided and licensed tool / algorithm or get a license to write their own tool.

Portable Network Graphics (PNG), JavaScript Object Notation (JSON) and vCard are some of the open data storage and transfer formats which have gained popularity for transferring different types of data between devices and applications. E.g. PNG is used to transfer and store images such as logos, vCard is used to transfer information about contacts & people (including their contact details such as email and phone number) and JSON has become the de facto format for transferring and storing any arbitrary information not covered by any of the standard formats. JSON is also THE standard format used by search engines to spider through a website's data to rank in search results. SEO optimization is all about getting the JSON meta-data correct. The JSON formats needed by search engines to identify different types of objects (e.g. a person, a music track, a tourist destination) is documented in a seperate standard called JSON-LD (JSON Linked Data).
Images courtesy: creativitywindow.com, encoding.com, gettinglegaldone.com

Conclusion

So as one can see, for free communication and data sharing among the participants of IoT, one needs to implement components which follow standards. With more adherence, more interoperability can be achieved.
In the blogs to come, I will discuss which open options are available at each level, so that one can implement a seamless, interoperable ecosystem of IoT devices.