Author: agup

Offline JavaScript Part 2 – Overview of Interfaces and APIs

In Part 1 we looked at the differences between partial and fully offline use cases. Part 2 provides an overview of the HTML5 Interfaces and JavaScript APIs that make it possible to go offline with web applications. Going offline involves working with multiple pieces and coding for specific patterns. I’ve tried my best to stick to technology that is widely available across the largest variety of browsers.

Offline dependencies

Offline web applications are dependent on three things.  It doesn’t matter if your application is partially or fully offline, you’ll still need to address these in your code.

  • Caching HTML, CSS and JavaScript
  • Data Storage
  • Offline/Online detection

Caching

Application Cache. The Application Cache, or AppCache, interface lets you specify and store HTML and CSS files as well as JavaScript libraries so that they are available from the browser’s native cache. Once an item is in the cache the browser will use it regardless of whether it’s online or offline. It’s almost like you never went offline!

The AppCache is an essential part of your application strategy for allowing offline browser reloads or restarts. Without this an application will simply fail to re-load while offline.

Data Storage

Browsers have a variety built-in JavaScript APIs for storing data. The data can be for maintaining the applications state such as for storing bookmarks and form data. Or, it can be used for storing information such as maps, address and phone lists, TO-DOs or points of interest for a vacation.

LocalStorage. The LocalStorage API is super-easy to use. It stores Strings in simple key/value pairs. It’s limited to about 5MBs on most browsers. The two main challenges you’ll run into with LocalStorage are hitting the storage limit and performance hits when serializing and deserializing data.

IndexedDB. IndexedDB is essentially an asynchronous noSQL database that lets you store a wide variety of datatypes so that you don’t have to deal with serialization/deserialization.  Datatypes include String, Object, Array, Blob, ArrayBuffer, Uint8Array and File. While many online sources will tell you that there isn’t a size limit, I’ll tell you that in general you should limit your storage on a mobile device around 50 – 100MBs to help prevent the browser from crashing.

WebSql. It’s widely recommended that you not build applications directly on WebSql. The World Wide Web Consortium (W3C) is letting this standard die off in favor of IndexedDB and LocalStorage. I’m really only including this here for reasons such as Safari 7 and and the Android native browsers before 4.4 only support WebSql. For more information on how to get around this read down to the section on IndexedDBShim.

3rd Party Browser Storage

If the built-in browser storage capabilities aren’t meeting your needs you still have other options.

IndexedDBShim. IndexedDBshim is a Polyfill for WebSQL-based browsers. Because IndexedDB isn’t natively supported on older versions Safari 7 and Opera you can use this 3rd party shim to transparently translate your IndexedDB code to work across Android and iOS.

PouchDB. PouchDB is an Open Source experimental library that is an attempt to smooth some of IndexedDB’s rough edges as well as provide additional functionality, such as the ability to sync with remote stores.

LocalForage (Mozilla).  LocalForage is also an attempt to bridge the gap between LocalStorage and IndexedDB. It gives you an interface that provides much wider browser coverage than IndexedDB by itself.  One of the downsides is the amount of storage you can use. If a user is on an older browser such as IE8 that’s limited to LocalStorage then that user will be limited to storing about 5MBs of data. If you requirements call for using more than that, such as downloading large address lists, then the app won’t work on that browser or you’ll have to build in some sort of paging mechanism that deletes the old data and brings in the new.

Offline/Online Detection

There are a number of ways to detect if the browser is online or offline as well as when the internet status changes.

NavigatorOnline.online.  Some browsers have a built-in detection mechanism. However, it is not always reliable and false positives are a distinct possibility. For that reason, you will have to build additional detection capabilities or lean towards a 3rd party library.

Offline.js. Offline.js is a small Open Source library (~3KB) that detects when you lose an internet connection and when it comes back up. While not perfect, it does handle a lot of cross-browser compatibility issues for you. And, if you find bugs you can always create a fix and submit pull requests.

References

Caniseuse – IndexedDB

Caniuse – LocalStorage

Caniuse – WebSQL

Let’s Take This Offline

Going Offline with HTML5 and JavaScript, Part 1

There are two primary use cases for going offline with mobile HTML5 web applications and JavaScript: partial offline and full offline. Before diving into building offline apps, understanding the differences between these use cases will help you build the best applications for your requirements. The functionality in modern browsers has finally gotten to the point where it is feasible (and fun!) to build offline web applications.

Partial Offline. Partial offline means the vast majority of the time the application is online, however it can continue to work if the internet connection gets temporarily interrupted. A partially offline app understands that requests for remote resources, that is resources that don’t exist on the device, will automatically defer to local resource, or at least fail gracefully, during the period of time when an internet connection doesn’t exist. Partially offline apps typically cannot be reloaded or restarted offline. The coding required to handle this scenario is much lighter weight than the architecture required for going fully offline. An example of partial offline is a reader app that pre-caches certain HTML pages of your choice. If the internet connection gets disrupted you can continue reading and navigating between the cached pages.

The partially offline scenario exists because there is no such thing as a perfect internet connection for mobile. In reality, internet connections and download speeds are very choppy when measured over a period of minutes or hours. Sure, some internet providers market 4G connections as being extremely fast, or have the best coverage etc., etc., blah. The bottom line is cellular and even WiFi internet connections are not guaranteed. A good example of this is coffee shops. They don’t come with an uptime guarantee, so if a couple of yahoos sitting next to you are streaming HD Netflix then that will surely bring the internet connection to its knees.

In reality, if you don’t live danger close to a cell phone tower and are moving around doing your job or running errands chances are your internet connection will fluctuate up and down over time. Anyone that owns a smartphone has experienced this at one time or another. Dropped calls are perfect example. You may be shopping and going in and out of buildings, or hanging out in the back of a taxi, sitting in your car pulled off the side of the road, or perhaps even just watching your kids as they play in the neighborhood park. A web application architected for partially offline will let you keep surfing or working for a short period of time, and hopefully long enough until the internet connection comes back up.

Full Offline. A fully offline JavaScript application is one that starts out online to download all the necessary data and files, then it can be completely disconnected from the internet. These apps can survive browser restarts and reloads, they can stay offline indefinitely and/or they can be resynced online at some point in the future.

Fully offline apps need to be architected in a much more robust way than their partially offline cousins. Partial offline apps can be considered more fragile than fully offline apps because they can’t be restarted or reloaded while offline, and you have to be very careful to limit their capabilities while offline otherwise the user can easily break the app. Fully offline apps are built with the knowledge that they may be offline for extended periods of time and need to be self-sufficient because users will be depending on them. If a fully offline app breaks then the user will be completely hosed (and very unhappy) until some point in the future where the internet connection can be restored and they can resync the app.

Offline apps can break in any number of interesting ways such as throwing a 404 when the user hits the back button or simply crashing when the app unsuccessfully attempts to a load a new page. By their very nature, fully offline apps may have larger and more complex data storage and life cycle requirements. They cache entire HTML web pages and all their associated JavaScript libraries, images and supporting data.

Examples of full offline apps include mapping apps, web email, reader apps, and any other apps that require information from a database.  User data is typically downloaded locally, stored on disk and then accessed by offline web applications. Any data that’s stored in memory will be lost when if the device or browser is restarted while offline.

 

Web offline versus Native offline

When building out your requirements, it’s a best practice to do an honest comparison between offline web capabilities and the offline capabilities of native SDKs.  In general as of the writing of this post, it’s fair to say that native apps still offer much more robust offline capabilities than the latest versions of mobile browsers. There are a few exceptions where browsers may have similar capabilities but almost always the level of control is more limited.

Native apps have the advantage because they basically have direct access to the device operating system and many of the capabilities are simply integrated into the respective SDKs. Here is a partial list of capabilities that are commonly seen in native offline requirements:

Web apps, on the other hand, run within the browser and are subject to any limitations imposed by the browser. The browser, itself, is a native app and it restricts it’s own children (web apps) to certain security restrictions. A few examples of web app limitations include:

  • Access to a limited number of censors. Access is not consistent across different browser types.
  • Limited control over location services via HTML5 Geolocation API.
  • JavaScript cannot programmatically read and write non-cached files on the device without user intervention.
  • Internet connectivity detection typically dependent on 3rd party libraries such as offline.js. Support is inconsistent across some platform/browser combinations.
  • Indirect and limited control over battery life and optimization.
  • Browsers and any of their associated tabs stop running as soon as the browser is minimized. If you have a requirement for the app to “wake up” from a minimized state under certain conditions you will have to go native.

 

Summary

Partial offline applications are design to continue working gracefully during intermittent interruptions in connectivity. Because offline is considered a temporary or even momentary condition in this use case, partial offline apps can use lighter weight architecture and have smaller data storage needs than full offline apps.

Fully offline apps are designed to be taken offline for extensive periods of time. They have to meet more demanding requirements and need a comprehensive architecture that enables storing of HTML files, JavaScript libraries, and user data as well as being able to handle browser reloads and restarts while offline.

Lastly, when having conversations about building offline apps you should weight web versus native offline capabilities against your requirements. Native SDKs still offer much richer control over most of the aspects of offline functionality.

Stay tuned for additional posts on this subject. Part 2 will look at the features and APIs you can use to take applications offline.

 

References

10 ways to deal with intermittent internet connections

How accurate is Android GPS? Part 1: Understanding Location Data

Wikipedia – Browser Security

Why I’m considering removing blog comments

Many others before me have disabled their blog comments. In fact, a blog post by Seth Goden from 2006 still sticks in my mind and all the outcry it generated from around the web. The longer my blog has been around the worst the spam has gotten. In a way that kinda makes sense as my domain name gets shared via spammer databases and my growing number of blog post links get stored in search engine results.

Spam comments are awful. As of today, Askimet says it has blocked over 250,000 spam comments since March 2014. During the time it took to write this post I’ve already received 16 spam messages. For every single legitimate comment, I get 1,000 spam comments. Seriously. If managed improperly, spam comments can crash your blog if your hosting provider limits the size of your blog database.

This happened to me earlier in the year. Silly me, I thought that it would be impossible for me to go over my previous hosting providers pre-set 2 gigabyte database size limit any time soon. Most of my blog posts are text based and take up very little room. The entire size of my blog database was around 15 MBs. I didn’t even know there was a problem until I couldn’t create any more blog posts. Upon further examination I had received over 75,000 spam comments over a 30 day period and they were waiting patiently for deletion. I’m not exactly clear on why Askimet didn’t correctly report these, or automatically delete them within 15 days, or what exactly happened. One thing was clear: my comments table in the database was full of tens of thousands of spam comments. I thought I had set them for automatic deletion but apparently Askimet only auto-deletes the worst of the worst spam and everything else goes to the spam queue. Caveat Emptor.

I still think I see the value of comments in that everyone including me can benefit from the “shared” feedback including myself. But, the increasing amount of time and headaches spent dealing with spam is making me seriously reconsider.