Using browser extensions to enhance user's browsing experience

What is a browser extension?

A browser extension is a piece of add-on software that is installed inside the browser and extends the browser's functionality. Addition of functionality can vary based on the browser. New functionality can be added at two levels:

1. To the current webpage: Based on the contents of the webpage currently loaded on the browser screen, an extension adds extra content to it that aids the user in some way and enhances his/her experience on the page. E.g. hyperlink all postal addresses on a page and offer to open them in a map if the user clicks on any of them.
2. To the browser as a whole: The browser itself exposes extra functionality which can be used on any page. Such functionality creates a new option on the browser's toolbar or menu or shows a new status on the status bar. E.g. one touch social media sharing button.

To explore exactly how extra functionality can be added to the browser, let us look at some real life examples of browser extensions.

Add to Pocket

Pocket is a popular service to store webpages for offline use, so that the user can read it later (in fact the tools was called Read It Later before it was renamed to Pocket). Since it is too tedious for the user to open up the Pocket website everytime s/he finds an interesting webpage, Pocket created a browser extension which creates a conveniently placed button on the toolbar. On clicking the button, the contents of the current page are automatically saved to the Pocket service without the user having to visit the Pocket website.

Read more about Pocket service here.

Buy Hatke

E-commerce is really big in India and the competition is cut-throat. Right now, there are at least 6 major players in the sector, with another 30-40 who are breathing down the necks of the top 6. The websites try to attract attention to each other by offering discounts and cashbacks all the time. As offers fly by and prices are slashed further and further, the user finds it hard to keep track of where s/he can find the best deals. Enter Buy Hatke. When the user searches for an item on site A, Buy Hatke finds out if there are special deals offered by competition. If so, then a small bar pops up at the top of the current page and the user can choose to navigate away to the website which offers the deal. At the time of purchase, Buy Hatke also looks for coupon codes offered by famous services like Groupon, which it tries to apply to the item being purchased and shaves off more from the final payable amount.

Read more about Buy Hatke extension here.


Search by Image



Google have always made web browsing easy with their highly relevant search results and images. However they have taken things a step further with 'search by image' feature. One can query by image. They have an extension that puts a small blue camera when the user hovers over an image on any website. Clicking on the camera takes the user to Google's search by image website and the user can search about what is the image. The example above shows how the user ends up with results for Bhutan when s/he clicked on the monastery image.

Read more about Google Search by Image here.

Why create / use browser extensions?

Over the last 5-6 years, users have begun to use smartphones and tablets to go with their desktops and laptops. They use desktops while at office or at home for involved work, leisurely surfing and in-depth research, for which a large screen offers maximum benefits. They use laptops for work and research on the move. Phones are used for information at a glance, messaging and chatting, context-aware usage based on GPS and sensors and for casual photography and recording. Content creators customise the experience of their offerings to suit different screen sizes and contexts by making rich websites, responsive websites, apps and widgets.

However onc use case has been ignored in the midst of all these multiple-screen experiences. Sometimes, the one thing that the user does NOT want is multiple screens. E.g. it is a big distraction for the user to switch between two tabs of a browser or to look alternatively between the computer screen and the phone screen. Sometimes the user would like everything combined into one integrated workspace instead of switching windows, tabs or devices. This is the pain that browser extensions address. We saw the example of Add to Pocket extension above. The beauty of that extension is that the user can send any webpage to the Pocket cloud instead of having to open the Pocket website and switching between tabs. The value of Buy Hatke is that the extension prompts about better offers without the user having to keep 20 tabs open to check prices in 20 different e-commerce sites, nor does s/he have to open the Groupon app on his/her phone to look for discounts. Google Search by image offers the user to query for images directly from any website without having to leave that website.

Where can I get browser extensions?

Browser extensions are specific to particular browsers and are not compatible with the others. The browser that supports by far the most number of extensions is Firefox, followed by Chrome. Safari and Opera have decent support for extensions, but Internet Explorer has very limited support for extensibility using add-ons. IE does however support the concept of browser 'plugin' which is a topic for a different post. But to give you an idea, installing IE plugins is more like installing software on your desktop with a seperate installer file, rather than an intuitive experience from the browser.

• To get extensions for the Chrome browser, one can check out Chrome Web Store.
• For Firefox, extensions can be installed from Firefox Add-ons
• For Safari extensions, head over to Safari Extensions
• Opera browser can be extended using add-ons on Opera Add-ons

How can I make a browser extension for my service?

You have made a cool online service XYZ. Now you want to make the user's life awesome. How can you go about building extensions for your service?

Each browser requires its own process for browser development. At the end of the day, the browser extension is built and submitted to the corresponding Web store / Add-ons store. However extensions can also be hosted on one's own website.

• To build extensions for Chrome, the tutorial is here.
• You can learn about starting to build a Firefox extension here.
• Safari Extensions Development Guide is your definitive guide to enhance your service's experience on Safari.
• This page has all the info that you need on making add-ons for Opera.

Conclusion

While a typical Internet savvy user spends most of his/her time on the smart phone and apps today, that time is not 100%. Some part of the day is spent actually sitting in front of a PC / Mac / laptop to get work and research done and switching between tabs and screens is definitely a pain. So, while not as hot in the market as mobile apps, browser extensions do have a massive fan following and the number of extension installs everyday does speak volumes about their utility.

We have all been solving problems for the user on the go using mobile technology. However, if you find that the next problem that you want to solve should target the sit-at-desk user using a browser, consider building a browser extension. He/she will be grateful to you for it. Probably you should make a 'thank you' button on the toolbar to go with your extension, to compose and send a 'thank you' email to your email address automatically.

How going Open Source empowers you: Part 2

In the last post How going Open Source empowers you: Part 1, we saw the advantages for someone who chooses to use open source software. In order to use open source software, someone must first decide to release such software. But the very nature of open source software is that the secret sauce, i.e. the source code is released along with the software. Does this make business sense?

How the world understands (assumes) business works

Let us first look at why common beliefs about business screams out a big NO to open source. In a traditional business model, there are certain goods and services in which the market supply is much higher that the market demand. Let us take e.g. burritos. An average metropolis has hundreds of dedicated Mexican restaurants. World's top cities like New York and London might in fact have more Mexican restaurants than Mexico City has. A person who wants to eat a burrito has practically hundreds of locations to choose from inside his/her own city. There must be five to ten locations within a stone's throw away. How can a Mexican outlet attract a customer's attention and retain him/her. By differentiating in taste. The most exotic or authentic taste wins. If in such a competitive environment, the restaurant gives away the recipe, it would be a business equivalent of a suicide. How then can there be benefits in open source?

Let us look at some of the points that despite the above argument have motivated companies / individuals to open source their offerings.

How open sourcing your offering empowers you

#1. Having an open source offering online is more valuable than the best written resume

This is the most unsung benefit of having an open source project online. Sure, one can always contribute to an existing open source project to make it better. But having a project of your own from the scratch is even more eye catching. College students who have released a project to the public along with the source code have stimulated the infatuation of powerhouses like Google, Microsoft, Cisco, Intel and scores of others. Let us look at some examples.

1. Linus Torvalds submitted Linux as a university project.
2. RSA algorithm, which forms the basis for data encryption and network security around the world today was a university project by three students whose names' first letters were assigned to the algorithm (Rivest, Shamir, Adleman).
3. Sun microsystems, purchased by Oracle about a decade ago, has a name based on a acronym SUN, which stands for Stanford University Networks. Their initial success was around the success of their operating system named Solaris, which was proprietary. But the foundations of Solaris were around many smaller modules released as university-level community supported modules.

These three are examples of really big projects which almost made the students' resume redundant.

#2. Not everyone can cook, given the recipe. You will still have substantial market share and in fact will be the pioneer

Duplication and competition is one of the biggest fears that businesses have. Let me put it this way. Whether or not a business shares its secrets, there will always be individuals and organisations who will try to copy its products and services and achieve a reasonable level of success. There will be a certain section of customers who will find that duplicate good enough for their needs, if it is priced aggressively in comparison to the original. If you walk along the street shops of any Asian city, there will be duplicate Phillips players, duplicate Burberry suits, duplicate Nike shoes and a lot of other duplicate wares. Do you think they do not sell? They do so like hot cakes. The prices are a tenth or even a twentieth of the original and there is a substantial number of takers for that price. But with daily use, one will notice that the duplicates behave like ... well ... duplicates!

Even if a dish's recipe is shared, a suit's weaving technique described in detail in a manual or a a piece of furniture's carpentry shot elaborately in a video, there is only a small section of individuals or businesses who can successfully replicate a process down to the last instruction to the same amount of success. The original source's reputation as the master is generally very safe.

What about goods and items that can be copied and resold? Movies, software and music for instance? In a world of 7 billion humans, it is hard to comb through all the piracy already rampant and to enforce laws against the miscreants. Pirated copies are happily exchanging hands either for a fraction of the price or for free. While not common in the west, Asian countries have long been indulging in sharing material through piracy. In fact, piracy is what makes a lot of otherwise very expensive music, movies and software visible to more eyes. If the quality is really good, these persons may consider paying for the next concert or the movie or the next version of the software.

With both cases considered, if a business releases its offering as open source, there is a fairly little chance that anyone who tries spin off his / her own version using the source will have the same level of success as the original. Likewise, opening up the source will allow the offering to get more eyeballs and there is a chance that those persons might consider getting the next offering directly from the original business. There is the added advantage that the business will have started a mass adoption of their offering and will be considered a pioneer, which might be due to #3.

#3. Opening up a product will open more opportunities for additional business much more rapidly than a closed product

Companies are always looking for ways to increase their income by selling side products such as accessories along with their main product, thus spawning a range of products. In the hair care industry, a shampoo is best accompanied by conditioners and hair oils.

How can opportunities like this be created in an open source offering? Let us consider our 'burger with a recipe' example from our previous post. Our fictitious cafe C had decided to offer the recipes for burgers along with the burgers. Let us assume that this spurred some customers to try the burgers at home. How can cafe C leverage the opportunity that they just opened up. Instead of looking at the customers as competition, they can look at them as allies. They can segregate the customers into groups. Some of the customers might not even know where to start. Cafe C can sell a starter kit which contains all the ingredients packed, chopped up and just ready to be cooked (Indian business iChef does exactly that).

At another level, some customers may need quality kitchenware to make the burgers taste as close to Cafe C's as possible. Cafe C can tie up with kitchen supply outlets and ask the customers to quote a coupon code to the outlet in return for a discount. The coupon code tells the outlet that this customer was referred to by Cafe C and they pay them a commission for every customer referred.

In general, an ecosystem can be created for completely different business models for Cafe C, just because their recipe is open sourced and they have customers who are willing to try to use that recipe.

After open sourcing Red Hat Enterprise Linux, Red Hat went out to the market to see how people are using their offering. They found two very promising use cases.

1. A large bulk of users were using Red Hat Linux to host their web servers. Linux as an operating system for web servers is legendary. But Linux alone is not enough for a computer to be a web server. Most of the web servers were powered by J2EE and used web services such as Tomcat to set up a J2EE environment. The users were complaining about how tedious the setup was and how Tomcat was not scaling enough for their needs. Red Hat saw the chance to build a better J2EE environment to accompany their Red Hat Linux operating system. Along came JBoss, which went on to become a very hit platform for web services.
   
   
   
2. With the release of Red Hat Linux came aspiring professionals who went on to become experts in the field of setting up, maintaining and securing Linux machines. But the experts were looking for some kind of accredition for their hard work. While Microsoft had certifications like MCSD (Microsoft Certified Software Developer) and Cisco had CCNA (Cisco Certified Network Administrator), Linux experts had no accolades to call their own. Red Hat jumped at the opportunity and started the RHCE (Red Hat Certified Engineer) accredition. Red Hat now certifies hundreds of thousands of RHCEs per year and earns millions of dollars as certification fees.
   
   
   

Conclusion

As you can see, open sourcing one's business offering may seem like a suicide at first, but after a careful study of how open source companies have done in the past 10-15 years, the case for business models using open source software is extremely compelling. After all, companies like Facebook, Google, Apache, Red Hat and Twitter have not done bad at the market at all. Each of these companies have made open sourcing a core principle in their way of operating and have found enough business models to turn into blockbuster hits for their share holders, customers and investors.

How going Open Source empowers you: Part 1

One of the most common buzzwords used in technology today is 'Open Source'. We keep hearing every now and then that a company A has made its offering open source. And on the other hand, we hear that a company B has switched its entire systems from proprietary solution X to open source solutions Y. And it is not tiny organisations that we are talking about. It is companies like Google, Oracle and Apache. Now before we see why those companies are using open source products and why they making their products open source, we need to understand what is open source.

Open source: When you give away the recipe along with the dish

Imagine that a cafe C's burgers are very popular. But imagine a twist in the story. While there is no doubt that the burgers sell because of their taste, a lot of people actually flock to the cafe to buy the burgers because along with the burger, they give away the detailed recipe inside the folds of the paper napkin. Cafe C does not mind if any of their customers replicates the recipe and goes on to make his or her own success in direct competition with C. Now image a woman W, whose milk shakes are a rage in her locality. Everytime she serves her guests some of her popular milk shake, they ask for the recipe and she willingly gives it to them. Since she serves milk shakes to guests who visit her home, she isn't charging for them.

Thus we see two categories of open source software. The cafe C's burgers are commercial Open Source Software, since one purchases a burger first in order to get the recipe, but the recipe will be part of the burger, no questions asked. Woman W's milk shake falls in a category called FLOSS or Free Libre Open Source Software, which is both open source and free of cost.
Linux by itself is a FLOSS, but Red Hat Enterprise Linux is a type of commercial OSS.

But wait, you may ask, if you give away your recipe, how can you make money? How does it make business sense to sow the seeds of your own competition? We will get back to OSS from the point of view of the open source producer in the next post. We shall first see how adopting open source software as a consumer empowers the adopter.

How open source software empowers the adopter

#1: Adopting open source lets you hike on your own thrilling path

Most of the hill stations in the world offer two ways in which you can go from the base of the mountain to the top.

One way is to take the national highway that starts from town B at the base of the mountain to the summit S which is the hill station at the top of the mountain. The path is paved and safe and user-friendly in the sense that the driver just have to follow the road wherever it takes them and does not have to worry about how to navigate to the top of the mountain. There are hotels, restaurants and motels along the way, making the tourist's experience very comfortable. The driver must follow lane disciple and must follow speed limits. Often the roads are narrow and winding and the driver is forbidden from stopping on scenic spots such as a beautiful waterfalls or if he/she spots some rare species on the road. The road may pass places where there are drop-dead gorgeous views across valleys where the driver's car cannot reach, because the highway authorities did not care to lay roads to that point. Occasionally, people who travel by the highway may write to the government to lay roads to spots where the tourists are dying to go, but the government may or may not relent, because of their own agenda. The government has set a bottomline for the highway, i.e. reaching from town B to hill station S. Everything else is out of their agenda and will not be given priority.

But there are some people who seek the thrill and adventure and simply refuse to use a highway which they find boring. They use a hiking trail which weaves through the heart of the mountain. They go at their own pace, shooting pictures, discovering new species, wandering off the trail to explore hidden spots and come away with an experience that thrills. On the other hand, they are left to their own faculties. There is no official route, so the hikers will have to pay attention to the trail. They may occasionally get lost and need a lot of patience to find their way back. They usually have to carry their own food and water. Since there is not official documentation on the trail, hikers usually reach out to hiker communities for their solutions.

It is the same with proprietary and open source software. Proprietary software is easy to use, but are stuck to a bottomline and are often not customisable. Requests to add or enhance features may or may not be entertained and if acknowledged, will be released along with the next version. This incurs a long wait. Open source software is flexible and customisable, but there is little or no documentation, so the strength of the community who use that software are often the best entities to reach out to when there is trouble.

#2: Adopting open source provides a lot of choices

Large food chains like McDonald's, KFC and Burger King have a very standard menu. There are probably about 15-20 items on their menu throughout the year and the menu has not changed for decades, save for a few seasonal variations. You can walk into one of these outlets and bet on the fact that the taste will be the same every single time. Amazingly, the taste is the same in almost every outlet in the world, with a few tiny variations in spiciness. The chains generally have a formula for how they make their items and guard it very secretively. Their standard global recipe and the resulting consistent taste are what make people coming back to them. We can debate the healthiness of the food, but there is no doubt that the taste experience that they have researched and come upon so secretly is so addictive! For a while! If you have the items in these outlets for more than a year, chances are that you will be bored and seek out some new experiences.  But, new experiences are the ONE thing that these chains do not have to offer. You either love their menu... or you dont!

Contrast this with eating noodles. As one of the most shared recipes in the world, noodle dishes have been imagined and re-imagined by thousands of eateries around the world. Packaged as Nestle's Maggi instant noodles, which cooks in just 2 minutes, served in the costliest of china clay in upscale outlets like Nooodle Bar (the three O's are intentional), sold in street food eateries in India, Thailand and Malaysia under various banners such as Hakka, Char Kway Teow and many different forms, noodles are as ubiquitous a fast food as anything can get. And they come with different textures and tastes. Long strands, thin strands, thick strands, tape-shaped, sour, hot and spicy, hot and sweet.... you name it, you get it. The main reason is that no single organisation withheld the recipe of noodles. Noodle recipes have passed many mouths, been written in many recipe books and have transcended multiple generations. In fact, so wide is the spread of noodle recipes that many street food eateries in India serve noodles steeped in Schezwan sauce as 'Chinese noodles', although the taste is quite Indian!

Open source software spreads in the same way. We have lost count of the number of different forms of Linux running in the world. Ubuntu, Fedora, SuSE, Debian just to name a few. The most famous form of Linux is of course Android. Android itself has been customised by various companies such as Sony, Samsung, LG, HTC and One Plus. In the end, the customer has a staggering variety of choices and he/she can pick what is best for him/her.

However it does mean that a user sometimes has way too many choices and as many communities. Each community will suggest using the variant that it follows. The user is left paralysed with choices even before he / she can start.

#3. Adopting open source frees the adopter from usage licenses and permits

Adding to 'the highway and the hike' example from #1, to use a highway, one must pay a toll. Some toll booths are happy to let you use the highway multiple times on the same day, while other toll booths mandate that a toll be paid every time the highway is used. The driver who drives on the highway must have a valid driver's license. To reduce pollution and congestion, some regions in the world also levy a pollution / congestion tax for entry of vehicles inside certain cities or towns or green zones. Most of these restrictions and fees are also on a per vehicle basis, i.e. the permits are issued based on a vehicle's license plate. So multiple vehicles will require multiple permits.

In contrast, a hiker can walk into his / her preferred hiking zone and camp for days together. Sure, there are special zones which require a small fee to be paid for permits from units like the forest department, but in general, you can just walk into hiking trails without having to think about any special permissions.

To use open source software, all that the user needs to do is download, install and use. There are no 'I agree to EULA (End User License Agreement which prohibits a user from sharing his / her copy of the software with others)' screens or any screens where product codes have to be keyed in. The user can use the software and share it with as many person he / she likes.

Conclusion

In conclusion, as an open source adopter, you can clearly see many advantages and some not so good points. The advantages definitely outweigh the pitfalls and you should certainly consider trying open source solutions for 2 months. Those 2 months will let you know if open source really works for you.

But as a business owner, does it really make sense to write your software and just give away its source code? Can you really make a sustainable business out of such a process? We shall see in the next post that it is very much possible to do so.

Should there be an app for that?

'There is an app for that'. Almost every user who uses a smartphone, especially an iPhone, has heard of this famous catch phrase made famous by Apple in 2009's World Wide Developer Conference. Why was the line so catchy and why did it become so famous? We have to look at phones before iPhone to analyse it.

The situation before the growth of apps

Before iPhone hit mainstream along with its iconic App Store, where one could get hold of apps to practically do anything that could be done on a smartphone, the ecosystem of app development and hosting it on a central repository managed by the phone makers / phone OS makers was very fragmented. Most of the time, the development environment was too complicated, it was very unwieldy to make a custom app and hosting the app for download was mainly the responsibility of the developers themselves and it was upto good SEO for an app to be spotted. Nokia did try to ease matters by introducing the Ovi Store, but anyone developing for Nokia using their Symbian platform and later on J2ME would agree that making an app needed a lot of skill, time and patience. The system was not as easy as iOS and Android are today. Blackberry's platform was slightly better, but their focus on business meant that their apps were centred around business and productivity tools and there was nothing by way of games or apps like Instagram, Whatsapp or any of the Google Apps in the Blackberry World. In summary, user who bought a Nokia or Blackberry were generally content with (or let us say, stuck with) the apps that were pre-installed in the phone during purchase.

With iOS and Android, Apple and Google focused exclusively on building an ecosystem where developers could easily build apps with powerful tools and APIs and upload them to the respective app repositories (App Store and Google Play) in a jiffy.

App explosion

Fast forward to July 2015. According to the numbers in statista.com, Google Play has 1.6 million apps and Apple App Store has 1.5 million apps. Now surely we cannot do a million and a half different things with a smartphone. Chances are that a typical user uses only between 2 to 15 distinct features on his/her smartphone. Typical uses would include Internet access, photography, geolocation, bluetooth connectivity and storage on SD card (a feature not supported by Apple anyway). Even if an app were to use a combination of the mentioned features, the number of combinations will run only into a few hundreds. So, it is clear that hundreds of apps are competing against each other over the same combinations of functionality. E.g. there are hundreds of photo snapping and sharing apps, hundreds of apps that deal with bluetooth sharing, etc.

Full-blooded competition

This means that when a company makes a new app, they may be actually competing with hundreds of apps which are achieving the same thing, although with their own set of conveniences and user experiences which draws a certain set of users to their app. But it does mean that those users will also install competing apps and may or may not stay loyal to any of the apps.

In fact, observations show that apart from the pre-built apps, a typical non-techie smartphone user has only 3 apps actively installed and being used from the app store. More techie users will generally have upto 15-20 apps. But both sets of users generally use only upto 7-8 apps on a regular basis, i.e. at least once a week.

With such user choosiness and such an oversupply of apps, the situation looks just like the scene from Finding Nemo where a flock of seagulls are chasing after a pelican to get their bite of morsel.

Cut-throat competition among apps means that they are constantly stuck in a cycle of having to grab the user's attention using marketing, promos and awareness tactics. When a new user wants to install apps, it looks like the apps are shouting out to the user like the seagulls in Finding Nemo!

Why you should care

The situation paints a grim picture for anyone who wants to roll out his/her own app. Is it really such that after putting in so much time and money on building an app, marketing and acquiring users, we can actually wake up one fine morning to find that users are actually jumping ship to a rival? The hard truth is YES. As long as an app is an also-ran among a pool of rival offerings, there is no guarantee that users will stick to the app unless it has something uniquely valuable to offer. And that value has to be really ground-breaking, disruptive and radical for someone to hang around. Or it must have enough fan-following and generate a humongous amount of content currently not being done so by the rivals.

The moment of truth

Companies and individuals must stop and think why they really need an app to achieve their objective. To put the question more specifically and to make things clearer:

What kind of functionality am I introducing into my product that is making me overlook a mobile enabled website and instead making me invest my time, money and resources into making one app per platform, i.e. Android, iOS and Windows Mobile?

What am I bringing into this product such that the convenience of writing a one-size-fits-all HTML5 has to be intentionally ignored and the hassle of writing 3 seperate codebases is the only option to achieve the means?

If your answer is, "Everyone has an app and I do not want to miss out on the mania" or "If I do not make this app and instead make a mobile website, my customers will think that I am antiquated", then relax. You may think that everyone including your customers and share-holders may care if you have an app or not. In reality, they couldn't care less as long as there is a solution that solves their problem and works 100%. It does not matter if it is an app or a website as long as the user has somewhere to turn to as a solution for his/her problems.

Giving HTML5 and responsive web a chance

Apart from the obvious fact that one codebase and one URL will work on any device, be it iOS, Android, Windows mobile, tablets, desktops, kiosks and a lot of HTML-compliant devices in the future, responsive web is super easy to design and code, needing much fewer lines than native code to achieve most tasks. With a plethora of frameworks now available, such as jQuery, Angular, React, Bootstrap, Meteor and Polymer, the feel of mobile web is getting closer and closer to a native experience. 

Browsers are getting faster, HTTP as a protocol is getting more streamlined with two-way connectivity using websockets, HTML5 and CSS3 are matching native apps special effect for special effect and animation for animation.

It would be a mistake not to consider making a responsive website to get a solution up and running as quickly as possible. If after careful consideration, app specific features are needed, then one must consider building an app.

To appify or not to appify

Fear not my friends, after all my lecture, I would like to end this post on a productive note. The following section lists some key functionalities of smartphones. Some of these features demand that an app be made to support them. For others, a responsive website will provide a sufficient and even better experience.

1. Text content editing with fallback offline storage: Think Google Drive and Evernote. If the data is text only, then look no further than responsive web. It is well capable of syncing with cloud to permanently store the data, while at the same time storing data offline just in case the user decides to document his trek from a mountain with no coverage. The only reason that Google Drive and Evernote had to make apps was reason 2, which is coming up.
2. Photography: Alas, on this one, the app wins. Responsive web has not yet found a way to use the camera.
3. Photo effects: Surprisingly, a responsive website is enough, provided photos are uploaded from the device instead of being taken directly from a camera. Pixlr is a good example.
4. Push notification: With the introduction of websockets, push notifications are now supported directly by responsive web. Facebook has integrated this seamlessly.
5. Drawing tools: One look at the website draw.io and you will never consider writing 3 seperate apps to replace your paper napkin diagrams.
6. Geolocation: GPS support has been in responsive web for almost 8 years now, ever since HTML5 made its grand entry. Google Maps has been pushing a pin on the top of your apartment on its web edition for several years now.
7. Sensors: Going for a jog? A swim perhaps? Sorry, a website won't help you here. Sensors are still off limits for responsive web and are expected to stay that way until W3C and IEEE join hands to form some standards on how to interact with sensors via web programming.
8. Shortcut on the phone's home screen: Answering this is a bit spotty. iOS has been supporting this feature for websites off the bat. Their Safari browser was beautifully integrated with their home screen. For Android, since Lollipop and with Chrome's latest mobile version, a website can set up an icon on the Android home page. But, hey wait, what if I love to use Opera. Well, in that case, things start to get tricky. To get the shortcut on the home screen, one MUST open the website on Safari (for iOS) or Chrome (for Android). But once the shortcut is placed, the website can be opened on any browser.
9. Completely offline content: Websites were not made to be consumed offline. Sure, one can always make a webpage with a barrage of links to download content to be viewed even when offline, but apps can be made such that they have offline content as soon as they are installed.
10. Audio / video streaming:  HTML5 introduced the support for playing audio and video without having to use memory guzzling Flash and Silverlight plugins. However they support only open standards and formats such as WebM and some popular formats like mp4, which is good for consuming publicly free to play videos. To play DRM encrypted / licensed content, special algorithms must be used. Their algorithms cannot be plugged into the browser on demand, but certainly can be integrated in an app when it is getting built.
11. Bluetooth: Bluetooth has been the domain of apps for quite sometime. However the introduction of Web Bluetooth and its support on every browser will bring this functionality to the browser. Currently it is a drafted specification and there are code sources for download to implement Bluetooth Low Energy, which is a lower energy version of Bluetooth, more favoured for tracking devices, rather than offering full functionality like file sharing and streaming. Full stack functionality and complete adoption by every browser will take time. Currently only Chrome supports web bluetooth in Low Energy mode.
12. Chat: Chat has been a browser-centric exercise even before the Google Talk era. Players like Yahoo chat and various dating and speed dating sites were websites before chatting apps for the desktop caught on.
13. Features requiring rapid changes: The odds are heavily stacked in favour of websites here. When features are nascent and need to be changed almost on a daily basis, websites can make sure that changes reach the users almost instantly. This is because the codebase of websites is present on the web servers of the product company rather than on the user's phone. With apps, it requires the user to find and trigger the installation of the latest version of the app. Sure, there are ways to force the user to update the app, but all methods have an element of friction and ultimately the user is the one who has to choose to download the latest version. With a website, the user doesn't have to opt to download or install anything. When s/he fires up the website during the next usage, the changes are automatically reflected by the browser.
14. Using protocols other than HTTP: The only way to go is to use apps. Browsers are inherently built to work on HTTP protocol. Period. For using any other protocol such as peer-to-peer networking like torrents, a browser simply does not have the support. A lot of protocols can be supported by a layer of HTTP, e.g. Email, streaming video calls, such that the browser can work with the HTTP layer and those HTTP messages are then converted into the underlying protocol messages, e.g. SMTP for email.

Behind technology: What seperates the winners from the also-rans

Hello Internet, in today's article we shall why in order to build long lasting solutions, we should keep aside the eternal arguments about what technology to use for building software and rather bring out a large whiteboard and first attempt to solve problems in a time tested way: brainstorming, diagrams and formulae.

Lasting companies and solutions: What seperates the best from the good

Let's play a small rapid fire game. Try to answer the following questions within seconds of reading.

1. You need to search for the recipe of a truffle cake. Which site will you head for?
2. You are standing at the airport and need a taxi to take you home. Which app do you reach for?
3. You have some awesome pictures from your visit to Thailand. Where do you upload your photos?

Chances are that you didn't think even for a second before answering. In fact, the answers might have hit your mind even before you read the question completely.
What makes solutions like Google search, YouTube, Uber, Facebook, Picasa, Flickr and Instagram so good at their game? It is very easy to conclude that technology is the ONE answer to such seamless solutions. However, technology is only the means to an end. The same technologies that are available to the top 10 apps / websites are also available to everyone else. In the great levelling playing field called the Internet, easily downloadable and commoditised technology is not what seperates the best from the good. These companies are in fact excelling at something that happens even before technology is applied, or even before the engineers finalise on what technology is best for the situation. It seems very contradictory, but these companies are good at sitting down together in front of a whiteboard, writing down points on a marker, drawing diagrams and paper models and playing out the entire solution with paper and pencil before they enter the technological world of bits, bytes, codes, apps, SaaS and the market.

  

Image courtesy: 123rf.com

But just what is it that they are drawing on whiteboards and modelling on paper before sitting down to write programs? And how does that make a difference to the final solution? As it turns out, they are figuring out something very fundamental to the world of computing. Regardless of whether you use Android, iOS, Java, Python or NodeJS, computing devices will understand only two building blocks. Enter data structures and algorithms.

Data structures and algorithms: What they are

Computing devices are made to store data over a long period of time. Since data storage is such a core functionality in computing, scientists and engineers have turned extensively to mathematics to study the various forms in which data can be represented, such as sets, lists, matrices and the like. Different types of data need to be stored in different forms and there is an entire subject behind the study of how efficiently to store data such that it can be accessed fast when required later. The form in which computing devices store data is called data structure.

In order to store data and fetch it for use later, a computer must go through a set of instructions which when given a piece of data must find the right place inside to data structure to put it and then later use another set of instructions to fetch data quickly from the structure again. Just like data structures, a carefully planned sequence of instructions goes a long way in ensuring that there are no major bottlenecks while retrieving data and that the retrieval is as quick as possible. Such as sequence of instructions is called algorithm.

A well planned combination of data structures and algorithms can mean the difference between a smooth and a frustrating experience for the user. Planning a good combination often requires engineers to abandon a computer for a week or several weeks and simply draw out diagrams and write formulae on a whiteboard or a piece of paper before writing the first line of code.

Using data structures and algorithms: A real life example

To really understand how the choice of data structures and algorithms affects user experience, let us take a real life example.

1. Let us say that Eddie has just moved into a new neighbourhood and is invited over to a get-together in his neighbourhood's club. Eddie takes along a sheet of paper and a pen, meets new people and notes down their names and numbers. He notes down names in the order in which he meets the persons and the names are not sorted in any order which makes sense to Eddie when he wants to look up specific names later. However a list works for him right now, since noting down is best done in the simplest way possible. This way, even if he meets three persons together and all of them rapidly dictate their names and numbers to him really fast, he can keep up, since he is simply appending new names to the list and his hand can instinctively reach the end of the list as soon as he hears a new name and number.
2. However the list is not sorted in any particular order that makes sense, so Eddie goes home and fishes out his phone book. The phone book is structured such that it is easy to find names in a snap. The names are arranged in alphabetic order, with pages having markers for alphabets. To find a name, Eddie just goes to the page which is marked with the alphabet with which the name of the contact he is seeking starts. E.g. to find the number of a friend named Carlos, Eddie simply goes to the page marked with letter C, on which he can find all his contacts with names starting with C. Carlos will be right up there.
   

   

   Image courtesy: amazon.com
3. Like any good affable person, Eddie wants to wish his friends on their birthdays and anniversaries. Let's say the day is 5th March. Alas, the phone book has no good way to pick out if any of his contacts has an event going for him / her that day. Even if he records each contact's birthday against his / her name, combing through each contact's record to find out if this day is special for him / her isn't a very attractive option. But Eddie is smart. He has a good calendar, where he marks the name of the person and the occasion related to the that person, against that date on the calendar. Eddie just looks up the date on the calendar and calls the person to make his / her day.
   

   

   Image courtesy: weknowyourdreams.com
4. It is the soccer world cup final and Eddie wants to invite over all his soccer loving friends to his home to watch the match on his big screen smart TV at night. However, not all his friends are into sports and not everyone likes to stay up at night to watch TV. Again, going through his phone book is not the best option to gather a list of people who like sports and do not mind staying up late. Luckily, Eddie has been fairly inquisitive while talking to his friends and has smartly made categorised lists on who likes soccer and who are night owls. He just needs to look at his categorised lists to find friends who would fit the occasion. He can look up the names under the categories and then use the phone book to find the phone number and then call each friend.
   

   

   Image courtesy: lifehack.org
5. Eddie decides to throw a barbecue party at his home's lawn. To make it personal, he decides to visit each house to invite his friends in person. He decides to bike around the neighbourhood and distribute hand drawn invitation cards to each family. But what route should he take to cover each family's home and do it in the shortest route possible? Will his phone book or calendar help here? Of course not. Perhaps a map of the neighbourhood will. And Eddie does have one. So he sets down to work, drawing a good route with a pencil over the map. Before long, the cards have been distributed and Eddie is set to have an awesome barbecue party.
   

   

   Image courtesy: 123rf.com

As you can see, Eddie has been paying attention to how he is going to use his contact list for different purposes and has prepared himself for these. By augmenting his contact list with a phone book, calendar, categorised lists and a map, he has given himself a very good chance of smoothly getting exactly the information he needs about his neighbourhood instantly. Had he missed out on any of the tools mentioned, it would have been a pain for him to get the right information. The different tools he used were identical to data structures and methods he used to retrieve the info are algorithms, e.g. turn to letter C on the phone book, look up 5th March on the calendar, draw a route on the map, etc.

The above illustration shows how having different representations of the same small neighbourhood and its people can drastically improve or destroy the experience of having to use that information. Imagine what it would be for Google or Uber to not use the right data structures and algorithms for the right purpose, with the humongous amount of data that they have.

Conclusion

Using the right data structures and algorithms seems to be common sense, but more too often, companies get caught up in the lure of technology and fail to pay enough attention to the basic building blocks and the core foundations of data structures and algorithms. Most companies in fact brag about having used cutting edge technologies such as their solutions being built on NoSQL, using web sockets or using cloud computing and we have very few companies that boast their algorithm can allow them to service 10000 users at once.

Since the inception of computing, data structures and algorithms have been unsung heroes and they still continue to be unsung, but that does not change the fact that they are heroes.

Measuring and Charting the progress

Hello Internet. We have seen how devices gather data from the field and send it over to the enormous cloud for storage. However stored data is of little use for a person who is trying to gather meaningful insight from it. The data sent over is raw data and is very difficult for the ordinary user to make sense. This is where querying and reporting comes in. Extraction of volumes of historic data and presenting them in the form of reports is what puts value into the painstaking collected data.
Here is a pyramid which shows data and how it is distilled into wisdom.

The basic strategy to convert huge data into wisdom: Divide and Conquer
Because of the sheer volume of data, it is smart to first break down the gathered data into manageable pieces and then convert those pieces into something meaningful. A single system generally has no means to crunch all the data in one bite on its own. Here are some of the strategies and technologies used to achieve conversion.

Crunch the data as it comes

This is the most preferred strategy when it comes to real time reports. The data that is coming in is crunched live into different types of outputs. Since the data generally comes in really rapidly (also called firehose input), the crunching system must have a high amount of processing power. Each input is generally processed into fifty or hundred types of real time metric, such a number of users currently online, number of active transactions, live incremental count of progress, etc. Also since these metrics are written live, it is a big NO-NO to store such rapidly changing data into disk storage such as hard disk. The live data must reside in faster memory such as solid state device or RAM. Multiple copies and backups are desired since such types of memory devices are typically temporary and lose the data when electricity is stopped or if the memory happens to be disconnected. Apart from hardware, it is necessary to use software which must be capable of accessing data & databases in-memory rather than from the disk. Traditional databases such as Oracle & MySQL do not have a good support for memory databases due to their heavy memory footprint and overhead. Enter Redis. Redis is one of the most successful in-memory data storage mechanisms and can support voluminous amounts of reads/writes per second. Other solutions such as Cassandra can make databases both on disk & in-memory.

While making real time metrics, it is important to plan out the metric well in advance since it would be very difficult to make a new metric on the fly due to the amount of crunching needed to bring the data upto date and then keep up with real time.
Another important point with real time data is that it must be simple enough to be derived in a very short time. E.g. Total online users at the moment is simpler to derive than total online users who are between 26-30 years of age residing in London. Not only does such conditional derived metric need more processing, but one would then have to make real time metrics for each such age group & city, which will quickly eat up the memory. Such reports are better off derived periodically and stored on a higher volume storage such as a disk to be fetched later.

Crunch data periodically

For periodic reports such as daily reports or monthly reports, this is the most efficient strategy. Since such reports are not required at real time, one can schedule them to be generated at intervals when the traffic is either very low or non-existent, such as midnight every day. Periodic data are generally summaries and do not need to store each & every field that is in the main transactional databases. For this reason, they are kept seperate from the transactional databases and even use different technologies such as hierarchy databases to show metrics grouped under various categories. The most common databases for periodic reports are MongoDB and Cassandra. The data is generally stored on disk or on a long term storage such as Amazon EBS, S3 or Glacier, which are cloud storage offerings, so that the business does not need to maintain its own storage disks for storing historic data.

Ad hoc reporting: Use multiple processing units to process blocks of data parallely

Despite all the careful planning, there will be instances when a business will need to generate reports based on adhoc requests. In such cases, it may be needed to call upon the business's entire history of data since the day it started operations. Such reporting is daunting and the business should ask for a reasonable amount of time to return the results to the stakeholder which raised the request. In the meantime, the business can use the following methodology to generate the report. Divide the data into manageable pieces. Set up a sizeable number of processing units to work on each piece. Start parallely processing each piece to gain outputs per piece. Gather the piecemeal output from each processor and join them up to get the final result. The above steps have one name in the industry: Map-Reduce. Mapping is the process of dividing and assigning pieces to processing units, whereas reduction is the process of gathering piecemeal results and combining them to get one result. The enormous data from which to obtain the result is called Big Data. The most popular technology used for Map-Reduce is currently Hadoop. Other company offering such Google Big Query and Amazon Elastic MapReduce are also gaining traction.

Over the past few blogs, we have seen how IoT works from the sensor data gathering phase to generating reports on the cloud. From the next post, we will see popular posts related to IoT, that I will curate from over the Internet.

Gathering and recording data from the field

We have seen in the previous blogs how devices on the field shoot out data to the cloud. We have also seen how the cloud can instruct the field to carry out remote instructions. We will now see what one uses on the cloud in order to set up the infrastructure to manage huge reams of field data.

What exactly is the cloud

Jargon speak has introduced a lot of words these days and the word 'cloud' is even more prevalent than 'IoT'. To super-simplify the definition, think of it this way. Instead of setting up our own infrastructure which would need 24x7 electricity, Internet connection, software and hardware maintenance and thus a humungous effort and a support team, we hand off all the headache to a provider who takes care of all these. This provider will make sure that he offers the following features:

• Server machines where our services will be hosted. The biggest feature is that server machines can be added quickly on the fly if the service's loads are taking a big hit during peak hours.
• Basic or complete suite of software which is needed for our services. Most providers will provide plans such as web hosting package or Web/Database combo package. Some of them allow customization to allow us to install our own software. Almost all of them have plans which set up the basics and then offer us a way to access the machines so that we can tweak them as we see fit.
• 24x7 availability with support. While a 100% availability is not guaranteed, most services go for 99 or 99.9% uptime.
• Payment plans which are generally very economical. Most of these services offer pay-as-much-as-you-use model. Almost all of them offer an advance lock-in for a few months or years at much cheaper rates if we commit to hosting our services on them for a long term. E.g. if the service costs 3 cents per hour of uptime, a service provide may agree at 2 cents per hour if we commit to hosting with them for the next 6 months.
• Dashboards to control, monitor and oversee the performance and tweak settings and add/remove servers on the fly.

The most popular cloud services at the moment are Amazon & Rackspace. Both of them offer Windows and Linux servers. Google App Engine and Google Cloud Computing have made huge inroads in the last 2 years with their super high performing and highly standards compliant offerings, but their solutions are Linux-only and hence there is no support for .NET based applications. Microsoft on the other hand offers the Windows-only Azure.

Image Courtesy: rackwareinc.com

Image Courtesy: digitalmunition.me

Image courtesy: cloud.google.com

Image courtesy: cloudtimes.org

This is by no means a complete list and there are many local and regional players in the game, such as Netmagic here in India. In Europe and USA, almost every Internet Service Provider also doubles up as a cloud provider.

What comes across from the field to the cloud

At the moment, there is no standardisation in the protocol or format with which data can be sent over from the field to the cloud. HTTP is the de facto protocol of choice since it has been around for far longer than the other protocols. Tried and tested, it is easy to get the man power to churn out a HTTP based data transfer service, since the World Wide Web uses the same protocol. However it is not uncommon to see other solutions such as Jabber (the most common chat protocol) or MQTT (the latest IoT specific data transfer protocol).

As far as the format of data goes, different vendors use different ways to represent the data. JSON is the most common format, followed by XML. However, many solutions use their own proprietary format, specially tailored for their IoT system and not interoperable with any of the others.

JSON and XML are particularly popular because of the ability to represent one piece of data inside another and thus achieve a hierarchy and an order of structure, which is how objects exist in the real world. However JSON has recently gained more popularity over XML since JSON can represent the same data in less number of bytes and is hence seen as network friendly for slow GPRS based networks. Some companies stick to XML though, since data represented in XML can be validated before being accepted on the server using a standard technique called DTD, without writing a line of software. No such standard standard validation technique exists for JSON and companies much implement their own validation mechanism by writing their own code / algorithm.

Typical JSON data showing employee records

Typical XML data showing book records

While JSON is receiving rapid adoption, there is no common JSON structure that is standardized across two vendors, thus defeating the purpose of JSON's interoperability across systems. E.g. In the employee structure above, one could split name as 'firstName' and 'lastName' by another solution. A system which is programmed to read 'name' would fail to recognise the split names.
To solve this, IoT has been increasingly tapping into a new standard called schema.org. This new standard has defined the JSON structure for many common type of 'things' such as an employee. Conforming to these pre-standardised structures will allow systems from various vendors to interoperate and synergistically work together.

Schema.org JSON structure standardised for a store. Any software which needs to represent a store should ideally follow this structure and that sofware will find it easy to work with stores all over the Internet.

How the cloud stores data

The data that comes from the field is fleeting and needs to be recorded somewhere for further analysis. There are several methods available to record and store the data. The obvious choice to do so is in databases. But there are so many types and brands of databases available that must be evaluated, compared and selected to arrive upon the best solution.

Broadly there are three main types of databases available these days: Traditional relational databases which have been around since the Darwinian days of digitally recorded data, hierarchical databases (typified by the term named 'NoSQL' these days) and linked databases (also called Graph).

Relational Databases

Avoiding a highly technical definition, let's just say that in relational databases, data is stored in tabular rows and columns, each table representing an entity or a 'thing', such as an employee. One table can link to another related table using a method called 'joining'. E.g. If there is one table storing all employee details such as his personal and contact details and another which stores his/her income and tax details, then for making a tax statement which shows the employee name and his/her tax liabilities, the two tables must be joined to fetch the info and make the report. Joining is computationally expensive and has been known to slow down data fetching operations. While relational databases are excellent for crunching transactional records into summaries very quickly, fetching data from multiple tables for the purpose of reporting is a massive bottleneck.

The most popular database solutions in the market currently are Oracle and MySQL. Microsoft based systems use Microsoft SQL server.

Hierarchical Databases

Borrowing from the concept of JSON described above, hierarchical databases use JSON structure to nest different types of data into each other, thus eliminating joins. Using hierarchy, it is possible e.g. a sensor's geographical details and its readings under the same structure, whereas in a relational database, both are required to be stored in different tables. This is excellent for generating reports where all the data is shown under one heading. More often than not, JSON-based databases use JavaScript as their language for manipulating data, instead of SQL which is used in relational databases. Hence the new jargon called NoSQL has been coined to describe hierarchical databases. Although number crunching is possible in hierarchical databases, relational databases with their clean tabular structure is more suited to such operations. It is not uncommon for solutions to use both relational and hierarchical at the same time. The relational databases are used to record data and crunch the numbers and later moved over to hierarchical databases for reports. The most popular hierarchy database used is MongoDB, since it was the first solution to attack the problem of hierarchy. Nowadays we have other solutions like Cassandra and CouchBase.

Linked Databases or Graphs

When relations between objects play a key role, e.g. for a social network which has relations among persons or for objects such as art or history, where one data is related to another and a researcher could go on to find related or similar pieces, neither table based data, nor hierarchy data fit the bill. One needs to store data in a form where a number of relations are traversed to go from point A to point B. E.g. in a logistics where a shipment must be delivered from Cape Town to Kyoto, one must be able to store and traverse data which represents something like truck from Cape Town to Johannesburg, flight from Johannesburg to Dubai, flight from Dubai to Tokyo and railway freight from Tokyo to Kyoto. Such structures are fulfilled by graph theory of mathematics and special database solutions exist for such problems. Graphs are represented by vertices and edges to inter-connect. Of particular advantage is the ability to assign costs or strengths (also called weights) to the connection between two points which helps in finding the most optimised solution for a given path.

Data represented as a linked database or a graph. The points (called vertices) are the individual objects such as cities, and the connections (also called edges) are the relations between objects, such as an available transport.

The most common database used for graph databases is the Google offering named Cayley. Other solutions such as FreeBase are also available.

Conclusion

This post has dissected the various options available for gathering and storing data as permanent storage records in the cloud. In the next post, we shall see what technologies exist to extract data from the databases once it grows into a monstrous size such as billions or trillions of records.