blog

HTML5 Application Cache study

By Julien Van Den Bossche

1. Intro

After running some tests on mobile cacheability, I led some new research on the HTML5 applicationCache on mobile. Using applicationCache could be a way to bypass the cacheability problems encountered on Iphone and to develop a high performance web application.

ApplicationCache allows your application to work offline by caching resources declared by the developer in a manifest file.

The official specs: http://dev.w3.org/html5/spec/offline.html#offline

2. How does it works?

2.1. Declare a manifest file

A manifest file, hosted on your webserver, should be referenced in your webpage by adding a "manifest" attribute to the HTML markup.

<html manifest="cache.manifest">

The content of a manifest file defines the caching policy of your website.

It should be served with the content-type text/cache-manifest.

2.2. Create a manifest file

Following the specifications we can declare 3 sections in a manifest file: CACHE, NETWORK and FALLBACK.

The CACHE section describes the resources which should be cached by the browser.
The NETWORK section describes the resources which must be addressed through the network only.
The FALLBACK section allows to declare resources to be displayed in case the browser is unable to retrieve the original ones on the network.

The first line of a manifest file should be CACHE MANIFEST. The following lines can be resources, comments or a new section.

CACHE MANIFEST
# This is a comment

img/myPic.png
myScript.js

NETWORK:
/
http://www.google.fr/images/srpr/nav_logo13.png
req/ajax.php

CACHE:
Img/myLogo.png
staticJson.js

FALLBACK:
img/ offline.png

Note: The HTML file which references the manifest file is automatically cached. The manifest file should not be cached, so don't forget to set the right http headers returned by your webserver (Pragma, Cache-Control, Expires, Etag).

2.3. Browser behavior

At the first access, the browser loads the web page, check if a reference to a manifest file is declared, and try to fetch it if so.

The manifest file is read and all the resources declared (in the "default", "cache" and "fallback" sections) are downloaded.

Note: if one resource is unreachable, the applicationCache process fails and the "old" version (the one previously stored in the cache) is displayed

At the second access, if the navigator is online, the browser loads the webpage from the cache and check if the manifest file has changed. If not, the resources declared as cacheable are fetched from the cache, others from the network.

If the manifest file has changed, the browser download all resources declared in the new manifest file.

Note: the manifest file is considered modified when its content is modified. You can just add a comment to change the manifest file.

2.4. Updating the cache

As I mentioned, updating the cache is automatically managed by the browser, if the browser is online and if the manifest has changed, the cache is automatically updated.

Note: to see changes, you have to reload the page to load new components in the applicationCache.

Nevertheless, developers can update the cache using the applicationCache JavaScript API.

ApplicationCache is available through JavaScript using the window.applicationCache Object.

You can update the cache using the update method of this object. Once the new resources are downloaded you have to swap the cache using the swapCache method.

Furthermore, you can listen to some events triggered by the applicationCache.


    Cache = window.applicationCache;
	
    Cache.addEventListener('updateready', cacheUpdateReadyListener, false);
    Cache.addEventListener('error', cacheErrorListener, false);
    Cache.addEventListener('checking', cacheCheckingListener, false);
    Cache.addEventListener('noupdate', cacheNoUpdateListener, false);
    Cache.addEventListener('downloading', cacheDownloadingListener, false);
    Cache.addEventListener('progress', cacheProgressListener, false);
    Cache.addEventListener('cached', cacheCachedListener, false);

     function cacheUpdateReadyListener(){
        	Cache.swapCache();
     }
….

Note: if the update fails the "old" version of the cache is used.

3.Tests

Iphone 3G OS 2.2
Iphone 3GS OS 3.1.2
Iphone 3GS OS 3.1.3
Iphone 3GS OS 4.0
Iphone 4 OS 4.0.1
Iphone 4 OS 4.3.3
Iphone 3G OS 4.2.1
Ipad OS 3.2
BlackBerry 9800 (BB6)
Nexus S (Android 2.3)
HTC Desire (Android 2.1 update 1)
Samsung I9000 (Android 2.2)
Orange Tactile (Android 2.1 update 1)
Samsung Wave GT-S8500 (Bada 1.0)
Samsund Wave GT-8530 (Bada 1.2)

3.1. First experiment: static webpage

I started my tests with a static web page which contains elements which are all cacheable.

a link tag in the HTML page loading a stylesheet
a link tag inserted with JavaScript
a script tag in the HTML page
a script tag inserted with JavaScript
an Iframe
an Iframe inserted with JavaScript
an XmlHttpRequest
an image tag
an image tag inserted dynamically
a video tag
an audio tag

<!doctype html>
<html manifest="cache.manifest">
    <head>
        <title>Cache Demo, static application</title>
        < link href="stylesheet.css rel="stylesheet" type="text/css" />
    </head>
    <body>
       <p>Application cache test on Iphone</p>
       <img src="img/cache.png" alt="Image from cache" />
       <iframe src="iframe.html"></iframe>
       <ul>
         <li><a href="javascript:Loader.loadCss('dynCss.css')">Load CSS</a></li>
         <li><a href="javascript:Loader.loadJs('dynScript.js')">Load JS</a></li>
         <li><a href="javascript:Loader.loadIframe('dynIframe.html')">Load Iframe</a></li>
         <li><a href="javascript:Loader.loadImage('dynImage.png')">Load Image</a></li>
         <li><a href="javascript:Loader.xhrCall('xhr.html')">Load XHR</a></li>
    </ul>
    </body>
</html>

Example of a HTML page declaring a manifest file

CACHE MANIFEST
# This is a comment, static application V0.1
img/cache.png

CACHE:
stylesheet.css

All the devices tested here, cached resources without any problem except for the audio and video files which are served partially by the webserver (The client sent an If-Range header).

Note: resources referenced in the head section of the HTML file are downloaded before the manifest file. Resources are downloaded twice if there are in the cache section of the manifest file.

3.2. Working with dynamic content

The second experiment was to test the network section.

I kept the same HTML page but declared some resources in the NETWORK section.

On each device the application works fine.

3.3. Working with external content

The third experiment was to test external content (hosted on a different domain than the webpage) in the CACHE and NETWORK section.

In each case all resources are displayed. Nevertheless content returned with partial content is fetched from the network whereas it was declared in the CACHE section (audio and video content).

3.4. Working with dynamic URL

The third experiment was to test dynamic URLs. For example, when you submit two different contents through a get parameter: http://mywebserver.com/index.php?action=search.

Iphone OS 2.2 didn't fetch dynamic resources whereas the other OS did.

If you want to use the applicationCache on OS 2.2 you have to list the whole exact URIs used by your application.

For the others OS, we need to declare at least the protocol and domain. For example declairing "http://mywebserver.com" is enough to reach URIs such as "http://mywebserver.com/path1/path2/myservice.php" from the network.

3.5. Working with wildcards in the network section

In the W3C offline spec I noticed that a wildcard could be available for the network section which prevent from listing all the resources which should be fetched from the network.

For this experiment, I replaced all the lines in the network section by the (*) wildcard.

Only the Ipad OS 3.2 and Iphone OS 4.x, Android 2.2, Android 2.3 and BB6 accepted the wildcard, displaying resources without any problem.

3.6. Fallback section

The FALLBACK section allows us to declare resources that will be displayed if the browser is unable to retrieve the original ones on the network.

Example:

FALLBACK:
/myOnlinePic.jpg /myOfflinePic.jpg

It worked on all devices except iOS 2.2.1 and Bada 1.0 and Bada 1.2. I also noticed that on iOS 3.1.2 and iOS 3.1.3 the fallback resource loaded slowly if the network call failed or if the online resource was unreachable.

On Bada phones, the test failed because of the update bug (see 'Updating the cache' section below)

3.7. Updating the cache

On all devices I tested the updating process by changing the manifest file.

On each device, excepted Samsung Wave GT-S8500 (Bada 1.0) and Samsung Wave GT-S8530 (Bada 1.2), the update process worked well.

On Samsung Wave GT-S8500 (Bada 1.0) and Samsung Wave GT-S8530 (Bada 1.2), once a first version of a website has been well cached, if we reload the site or use the JavaScript update method, the browser doesn't even try to reach the manifest file on our server, so we can’t update the website.

If the update process fails (Cache error event), the Window.applicationCache.update() doesn't work on BlackBerry and Bada.

3.8. Cache limit per webapp

3.8.1 Android

For all the android devices, it appeared that the cache limit size depends on the memory available when the applicationCache process proceeded. With an empty cache and a freshly rebooted phone, I managed to store more than 50Mb on a Nexus S (android 2.3) for instance. After a long time using the device, I only managed to store 15Mb (with this same phone), even if I cleared the browser's cache.

With the Samsung I9000, Orange tactile, HTC Desire, even if in some conditions I managed to cache more, most of the time I reached a limit of 5-7Mb.

Furthermore, with this memory size limitation, the update process can fail because if an application initially stores a total amount of resources close to the memory size limit, the new resources can’t be downloaded.

During the update process (when the manifest changes), the browser first download the new resources and then if everything is ok, swap the cache and then delete the old resources. During this downloading phase, the newly downloaded resources are added to the existing ones, so the limit size "counter" doesn’t start from 0 but from the current size of the cache.

To be clear let’s explain this with an example:

First, in a manifest file, lets declare 3 resources of 1Mb (so a total amount of 3Mb). During the first load, everything is OK, all resources are put in the cache, the cache limit is not reached (considering a total cache size limit of 5Mb).


CACHE MANIFEST
# v0.1

img/img1Mb_1.png
img/img1Mb_2.png
img/img1Mb_3.png

NETWORK:
*

FALLBACK:
img/ offline.png

Then lets change the manifest which still declares 3Mb of resources


CACHE MANIFEST
# v0.2

img/img1Mb_1.png
img/img1Mb_2.png
img/AnotherImg1Mb_3.png

NETWORK:
*

FALLBACK:
img/ offline.png

During the update process, the browser downloads the first two new images. This download is apparently stored in the cache area which is already populated by the actual 3Mb of cached resources, so the 5Mb limit is reached (3 + 2) and when the browser tries to download the “img/AnotherImg1Mb_3.png” file it just fails.

On the Orange tactile (Android 2.1 update1), when a cache error occurs due to too large sizes, the phone displays a message on top of the screen (no enough space for browser memory).

When a cache error occurrs, the JavaScript method Window.applicationCache.update() doesn't work on Blackberry and Bada

3.8.2. Iphone

For all the devices, except for iOS 4.x, the cache limit is 5Mb for all the resources. A single resource can't exceed 5Mb.

For Iphone 4.0 and 4.01, after a first download of components less or equal to 5Mb we can change our manifest file to download more components in the limit of 5Mb of new resources at each update.

For Iphone with OS 4.2x and greater, if the resources exceed 5Mb a message is displayed on the screen to ask the user to increase the storage. Depending on the size of the downloaded components, the proposal can be 10, 25 or 50Mb (maybe more but not tested).

Unfortunately, I noticed a bug on the Iphone: While accepting to increase the storage size, a cache error appears and the download step is stopped. If I reload the application, the new cache size limit is taken into account and there is no more cache error.

3.8.3. Bada

On Bada 1.0 and 1.2, I didn't run the test because I did not manage to update the cache.

3.8.3. BlackBerry

On Blackberry 9800, I didn’t manage to cache more than 5Mb for a webapp

3.9. Total cache limit

3.9.1. Android

On Android devices, the cache limit depends on the memory available on the phone as I mentioned on the previous test.

Application cache usees a Sqlite database which stores data on the file system.

Using the emulator, you can find the ApplicationCache database under /data/data/com.android.browser/app_appcache/ApplicationCache.db

When I did my tests with the emulator, when a cache error occurred, the sqlite returned an error ‘code number 13’ which means that insertion failed because database is full (see sqlite doc : http://www.sqlite.org/capi3.html) and which means that /data/data… is full.

3.9.2. Iphone

On Iphone, sqlite is also used to store the applicationCache data.

I didn’t manage to reach the cache limit size but for sure there is a limit. I managed to store more than 20 websites which declared 5Mb of resources each.

3.9.3. Bada

On Bada 1.0 and 1.2, I didn't run the test because I did not manage to update the cache.

3.9.4. Blackberry

On the BB9800, I managed to cache roughely 15Mb (7 websites with more than 2Mb of resources)

First change the manifest to declare 0 resources in the Cache section
Update (refresh button or window.applicationCache.update())
Change your manifest to declare the resources you want to cache
Update your app
If it doesn’t work, clear the browser cache and retry

3.10. Running a webapp from the homescreen

The last experiment was to know if the application behaved the same way when we bookmarked it on the home screen.

I noticed that the cache was shared between the two modes: launch directly from safari or launch directly from the home screen.

I also noticed a difference of behavior concerning the xmlHttpRequest. If the webapp is launched from the home screen the xhr returns "0" for the status (xhr.status) when the resource is fetched from the cache.

4. Tests summary

OS/Browser	Cache section	Cache external ressources	Network section	Fallback section	Allow external in network section	Allow dynamic URL	Allow wildcard	Cache limit per component	Cache limit per webapp	Cache limit
3G / OS2.2	OK	OK	OK	KO	OK	KO	KO	5 Mb	5 Mb	?
3Gs / OS3.1.2	OK	OK	OK	OK but slow	OK	OK	KO	5 Mb	5 Mb	?
3Gs / OS3.1.3	OK	OK	OK	OK but slow	OK	OK	KO	5 Mb	5 Mb	?
3Gs / OS4.0	OK	OK	OK	OK	OK	OK	OK	5 Mb	5 Mb	?
3G / OS4.2.1	OK	OK	OK	OK	OK	OK	OK	5 Mb	5 Mb	?
4G / OS4.3.3	OK	OK	OK	OK	OK	OK	OK	5 Mb	5 Mb	?
Ipad 3.2	OK	OK	OK	OK	OK	OK	OK	5 Mb	5 Mb	?
BB 9800 (6.0)	OK	OK	OK	OK	OK	OK	OK	5 Mb	5 Mb	~15Mb
Samsung Wave GT 8500(Bada 1.0)	OK but buggy	OK but buggy	OK but buggy	KO buggy	OK but buggy	OK but buggy	OK but buggy	?	?	?
Samsung Wave GT 8530(Bada 1.2)	OK but buggy	OK but buggy	OK but buggy	KO buggy	OK but buggy	OK but buggy	OK but buggy	?	?	?
Samsung I9000(Android2.2)	OK	OK	OK	OK	OK	OK	OK	Memory available	Memory available	Memory available
HTC Desire (Android2.1 update1)	OK	OK	OK	OK	OK	OK	KO	Memory available	Memory available	Memory available
Orange tactile(Android2.1 update 1)	OK	OK	OK	OK	OK	OK	KO	Memory available	Memory available	Memory available

5. Conclusion

Caching resources on mobile is mandatory if we want to offer a new experience on the mobile web. the ApplicationCache is a solution to bypass the native cache problems on Iphone (see previous study).

Iphone OS 4.x and Ipad OS 3.2 are ready for offline applications. Developers should use it because it enhances the loading time of any application and allow having a behavior very close to the one of the native applications. The ApplicationCache can be used on OS 3.1.x only if we can list all the hostnames of online resources asked by the webapp (network section)

Concerning OS2, ApplicationCache can be useful if we can define the whole list of URI asked by the cache. "Static" application can use it.

On Bada phones, even if the first load stores resources in the cache, it’s impossible to update the website, so do not use it for your website.

On Android Phones, greater than 2.1, implementation seems ok but the memory limitation, which can’t be known by the developer avoid him to be sure the webapp will be cached or updated.

BlackBerry 6 seems ok to use it but as Android the main difficulty is too manage the case when memory is full and so when the update process is made impossible. For the moment I did not find an easy and transparent way to bypass this limitation.

An alternative solution is to use the local database to cache resources. I developed a component under Winktoolkit which implements this solution : easyCaching (http://www.winktoolkit.org/documentation/module/714/ ).

Some studies on the database storage size limitations should be done soon to validate this kind of solution.

As we have problems with comments on our Wordpress, please leave comments via the forum or email me directly julien.vandenbossche__a-t__orange-ftgroup.com