Tuesday 14 December 2010

River of News, consuming a JSONP data feed


With the release of JQUERY 1.5 this post is now out of date, also Dave is updating his JSONP feed to make consuming of the feed easier. Once this is completed I'll update the blog with a new post on, but for now see post on River of News news group .

Recently Dave Winer, a pioneer of RSS, published a river of news feed via JSONP. This article shows how to consume such a feed and the pitfalls of JSONP. For starters check out the demo at http://martimedia.com/news.htm


To create the page I made use of the following JavaScript libraries

  • jQuery – Makes writing clean / cross browser JavaScript a breeze
  • jQuery.templates –Separates the presentation (HTML) from code(js)
  • jQuery.jsonp – Provides jsonp error handling

Step 1 – Download the news feed

In the past I’ve just used the jQuery $.get command to download the feed, but luckly, Dave was having teething problems causing timeouts and formatting errors. But $.get wasn’t returning any feedback on these errors. After a bit of goggling I found the following article on StackOverflow which indicates that JSONP requests don’t make use of XHR hence lack of error handling. To overcome this problem jQuery.jsonp is recommended. This is similar to $.get but supports error handling for timeouts and invalid JSONP.

   1: $.jsonp({
   2:               dataType: 'jsonp',
   3:               timeout: 10000,
   4:               url: 'http://jsonp.scripting.com/daveRiver.json?callback=?',
   5:               success: function (data,status) {
   6:                     ProcessData(data);
   7:               },
   9:               error: function(XHR, textStatus, errorThrown){
  10:                       if ( textStatus == "timeout")
  11:                           $("#newsFeed").text("Sorry, your request to scripting.com has timed out, please try again later");
  12:                       else if ( textStatus == "error")
  13:                           $("#newsFeed").text("Sorry, response from scripting.com is corrupted, please try again later");
  14:                   }
  15:       });

Step 2 – Process the news feed

Typically we would now write a load of jQuery to create HTML response, but in the new world of jQuery.templates we’re going to separate presentation from the code. This makes the code a lot cleaner and easier for others to reuse. In our example we’re not going to do any processing of the data, but just bind the data to the jQuery template with the following code:

   1: $("#newsFeed").html("");
   2: $("#newsTmpl").tmpl(data.updatedFeeds).appendTo("#newsFeed");

Step 3 – Generate HTML via a template

If you’re not familiar with templates then check out encosia blog . For Dave’s feed we have an array of feeds, which contain one or more stories. To keep us on our toes, Dave returns an object if the feed contains 1 item or an array for 2 or more items.

   1: <script id="newsTmpl" type="text/x-jquery-tmpl">
   3:   {{each(f,feed) updatedFeed}}
   4:       <h2>${feed.feedTitle}</h2>
   5:       {{if feed.item.length > 1 }}
   6:       {{each(i,item) feed.item}}
   8:                   <div class='item'>
   9:                   <span class='date'>${format_date(item.pubDate)}</span> ${item.title} 
  10:                   <a href='${item.link}'>${get_hostname_from_url(item.link)}</a><br/><br/>
  11:                   </div>
  13:           {{/each}}
  14:       {{else}}
  15:                   <div class='item'>
  16:                   <span class='date'>${format_date(feed.item.pubDate)}</span> ${feed.item.title} 
  17:                   <a href='${feed.item.link}'>${get_hostname_from_url(feed.item.link)}</a><br/><br/>
  18:                   </div>
  19:       {{/if}}
  21:   {{/each}}

Step 4 – Some formatting

To make it look pretty I made use of the colorzilla gradient editor , and a couple of helper functions to format the date and hostname.

Putting it all together in less than 100 lines of CSS/HTML/JS we have …..

   1: <!DOCTYPE html>
   2: <html >
   4: <head>
   5:     <title>River of News</title>
   6:     <meta name="Description" content="Example of consuming a JSONP data feed (http://scripting.com) via jQuery and jQuery Templates." /> 
   8:     <link rel="stylesheet" type="text/css" href="http://yui.yahooapis.com/3.2.0/build/cssfonts/fonts-min.css">
  10:     <style type="text/css">
  11:     body { /* generated via http://www.colorzilla.com/gradient-editor/ */
  12:     background: #87e0fd; /* old browsers */
  13:     background: -moz-linear-gradient(left, #87e0fd 0%, #53cbf1 40%, #05abe0 100%); /* firefox */
  14:     background: -webkit-gradient(linear, left top, right top, color-stop(0%,#87e0fd), color-stop(40%,#53cbf1), color-stop(100%,#05abe0)); /* webkit */
  15:     filter: progid:DXImageTransform.Microsoft.gradient( startColorstr='#87e0fd', endColorstr='#05abe0',GradientType=1 ); /* ie */
  16:     }
  18:     #page { width:800px; padding:20px 60px 20px 60px; margin:60px; min-height:600px; background:white; }
  19:     .date { font-size:80%; color:#888; vertical-align:top; }
  20:     #footer { padding-top:20px; }
  21:     </style>
  22: </head>
  24: <body>
  26: <div id="page" >
  28:    <h1>River of News</h1>
  29:    <p>Powered by Dave Winer <a href="http://scripting.com/stories/2010/12/06/innovationRiverOfNewsInJso.html">river of news</a> feed</p>
  30:    <hr />
  32:        <div id='newsFeed'>Loading news from scripting.com, please wait.....
  33:        <!-- news feed via template inserted here -->
  34:        </div>
  36:     <script id="newsTmpl" type="text/x-jquery-tmpl">
   3:     {{each(f,feed) updatedFeed}}
   4:         <h2>${feed.feedTitle}</h2>
   5:         {{if feed.item.length > 1 }}
   6:         {{each(i,item) feed.item}}
   7:                     <div class='item'>
   8:                     <span class='date'>${format_date(item.pubDate)}</span> ${item.title} 
   9:                     <a href='${item.link}'>${get_hostname_from_url(item.link)}</a><br/><br/>
  10:                     </div>
  12:             {{/each}}
  13:         {{else}}
  14:                     <div class='item'>
  15:                     <span class='date'>${format_date(feed.item.pubDate)}</span> ${feed.item.title} 
  16:                     <a href='${feed.item.link}'>${get_hostname_from_url(feed.item.link)}</a><br/><br/>
  17:                     </div>
  18:         {{/if}}
  20:     {{/each}}
   2:     <hr />
   3:     <p id=footer>Check out my <a href="http://martimedia.blogspot.com/">martimedia blog</a>, 
   4:     or my company website at <a href=http://martimedia.com/>martimedia.com</a></p>
   5: </div>
   7: </body>
   9: <script type="text/javascript" src="http://ajax.microsoft.com/ajax/jquery/jquery-1.4.2.min.js">
   1: </script> 
   2: <script type="text/javascript" src="http://ajax.microsoft.com/ajax/jquery.templates/beta1/jquery.tmpl.min.js">
   1: </script>
   2: <script type="text/javascript" src="http://jquery-jsonp.googlecode.com/files/jquery.jsonp-1.0.4.min.js">
   1: </script>
   3: <script type="text/javascript">
   5:     $(function () { 
   6:         $.jsonp({
   7:                 dataType: 'jsonp',
   8:                 timeout: 10000,
   9:                 url: 'http://jsonp.scripting.com/daveRiver.json?callback=?',
  10:                 success: function (data,status) {
  11:                         $("#newsFeed").html("");
  12:                         $("#newsTmpl").tmpl(data.updatedFeeds).appendTo("#newsFeed");
  13:                 },
  14:                 error: function(XHR, textStatus, errorThrown){
  15:                         if ( textStatus == "timeout")
  16:                             $("#newsFeed").text("Sorry, your request to scripting.com has timed out, please try again later");
  17:                         else if ( textStatus == "error")
  18:                             $("#newsFeed").text("Sorry, response from scripting.com is corrupted, please try again later");
  19:                         else
  20:                             $("#newsFeed").text("Error, status: " + textStatus + ", please try again later" );
  21:                     },
  22:         });
  24:     });
  26:     function format_date(date) {
  27:        var newDate = new Date( Date.parse(date));
  28:        return newDate.toLocaleTimeString();
  29:     }
  31:     function get_hostname_from_url(url) {
  32:         url = url.match(/:\/\/(.[^/]+)/)[1];
  33:         url = url.replace("www.","").replace("feedproxy.","").replace("feeds.","");
  34:         return url;
  35:     }
  38: </html>

Feel free to use the code to create your own river of news, I just ask you to post your examples in the comments section below.

Tuesday 7 September 2010

Google fusion tables

In the past I’ve created a few data visualisations using Silverlight, but though for smaller datasets a HTML solution would be preferable. Now with the help of Google fusion tables I can create simple datasets with custom info windows without the need to write any code.

In the example above we’ve created a Google Map showing expenditure of Primary Care Trusts(PCT) across the England, when you click on a PCT it will display a custom tooltip.


To create this visualisation without any code I carried out the following 6 steps:

Step 1 – Grab the data

For this example we’ve taken data shared by the Guardian Data Blog as a Google Spreadsheet, this is downloaded as an excel spreadsheet, which I removed rows such as the spreadsheet title and “regional total” rows.

Step 2 – Add geo hinting

To help Google geo-code the data I took the PCT name e.g. “Barnsley PCT” and region “Yorkshire and the Humber” to create a location column e.g. “Barnsley,Yorkshire and the Humber,England”

Without the hinting Google will assume the location is in the USA.

Step 3 – Upload to Google

Now with the geocoding hints in place we can upload the data, label the columns (important for step 5) and tell Google which column to use for location.

Step 4 – Geocode data

To geocode the data simply select the “visualize” menu bar and select “map” this will geo-code the locations to lat/long points.

Step 5 – Configure info window and styles

Now we have a map with lots of small dots for each PCT, choose “Configure styles” to change your pin style, and “Configure info window” to personalise the info window. For my demo I chose the custom template based on a mixture of HTML and meta data for field place holders.


Step 6 – Share and embed

Finally click on the “Share” button and select Visibility options Unlisted or public to share the data. Once the data is public we can use “Get Embeddable link” to get the HTML to embed in your website / blog

   1: <iframe width="500px" height="300px" scrolling="no"  
   2: src="http://tables.googlelabs.com/embedviz?viz=MAP&q=select+col0%2Ccol1%2Ccol2%2Ccol3%2Ccol4%2Ccol5%2Ccol6%2Ccol7+from+247676+&h=false&lat=51.78483389373529&lng=-1.197509765625&z=8&t=1&l=col3">
   4: </iframe>

Note: We can also export the data to KML, which could then be used in your preferred mapping solution, be it Bing Maps for Silverlight, Google Earth etc..

Thursday 8 July 2010

Part 3 of 3 - Creating a Custom Silverlight Pivot View application


This series of articles cover the process from concept to release of how RoomSeeker.eu integrated visual search using Pivot Viewer for Silverlight:

Part 1 – What is Pivot?

Part 2 – Creating a Pivot Collection

Part 3 – Creating a custom Silverlight Pivot View application (this article)

Part 3 of 3  Pivot View Silverlight App, Deployment and Summary

Create Silverlight Pivot App

The screen shot below shows how we’ve integrated a Pivot collection based on the Silverlight Pivot Viewer Control into our site in the form of a visual search page, you can try it out here.


RoomSeeker.eu Visual Search page

The app is a simple host for the Pivot Silverlight Control, with the addition of some buttons in the top right to select the collection ( East, South East, South West England etc.) and a back button in the left hand side to enable the user to see a previous query.

When the user zooms in we’ve added two custom buttons in the top right hand corner

  • Hotel Rates & Availability – Navigate the user to hotel info web page
  • Hotels nearby – Navigate user to hotels for the given tow (Histon in this case)



Even though the app is quite basic and probably needs a few more iterations, it’s already provided useful for queries such as:

  • “Show me all the AA rated hotels that take pets and cost less than £150 in the South East of England”
  • “Show me all beach hotels group by town in Suffolk”


Future improvements:

  • Deep linking – Currently I cannot share links via social networks e.g. click here for a list of golfing hotels in Suffolk
  • Browser navigation button integration – Make use of browsers back and forward buttons to flip between filters.
  • Dynamic pivot collections to overcome regional search limitations e.g. “Show me all hotels in England with golf courses”
  • Silverlight hotel information pages, rather than navigating to a separate web page
  • Analytics
  • Improved search panel
  • Custom loading page – % loading plus some help
  • Help for first time user, online video help (would be good if Microsoft had an intro to Pivot video we could embed in the app)
  • Use of MEF for custom search / branding options

Issues: Even though the app doesn’t do a lot the Pivot control and it’s dependences (including System.Xml.Serialization) result in a 800K xap file (2.5MB unzipped!). I’m sure this could be reduced by removing the decency on the Silverlight 4 Toolkit (April 2010).




Creating the deep zoom images is very CPU/IO intensive so these tasks were performed on a local PC. The files were then zipped up and uploaded to the server and unzipped. The pivot metadata is relatively small, so could be created either locally or on the server. On IIS I made the following changes

  • Add text/xml mime types for .cxml , .dzc , .dzi
  • Add compression for .cxml , dzc , .dzi
  • Add never expires HTTP headers for deepzoom folders

Note: If you client and collections are on different domains you will need to add a clientaccesspolicy.xml file to the root directory of hosting the collection.


Creating a prototype solution was quick and simple, by refactoring the NetFlix demo code from blog.smartx.com , using basic hotel photos with some simple search criteria.

But with most things it’s the final 20% that takes the time. Creating the composite images to include the hotels location on a map, getting the balance right on image quality vs download size, to what information to display on screen, and colours all took longer than expected.

Then the design of the search panel also took some time, deciding what data to display and how to group the data. This still isn’t perfect and will probably be refined over the coming weeks.

Finally the images are currently hosted on the west coast of America, this isn’t ideal as most of my customers are in the UK, so will probably need to look into use of a CDN / UK hosting to improve download speeds, possibly hosting on multiple sub domains to increase concurrent downloads and optimisation of the DeepZoom collections to share common low res images (e.g. generic green , yellow , blue images for images less than 16 pixels would reduce the file count significantly).

The Silverlight app is very basic, providing a host for the Pivot Silverlight control and directing the user to a RoomSeeker web page. In the future would be good to extend the app to support dynamic collections based on room availability.


This is part 3 of 3:

Part 1 – What is Pivot?

Part 2 – Creating a Pivot Collection

Part 3 – Creating a custom Silverlight Pivot View application (this article)

Wednesday 7 July 2010

Part 2 of 3 - Creating a Pivot Collection


This series of articles cover the process from concept to release of how RoomSeeker.eu integrated visual search using Pivot Viewer for Silverlight:

This is part 1 of a 3 part series:

Part 1 – What is Pivot

Part 2 – Creating a Pivot Collection (this article)

Part 3 – Creating a custom Silverlight Pivot View application

Creating a Pivot View Collection

For a deep dive on general collection design check out the article on silverlght.net, in this post we look at the steps required to create a pivot collection.

  1. Create composite images for input to deep zoom
  2. Create deep zoom data
  3. Create Pivot meta data

Step 1 - Create composite images

To keep things simple, pivot view only displays static images, so any text overlays, borders etc need to be burnt into an image.

Key things to consider are aspect ratio, size, borders, and image quality along with the information you want to overlay. For RoomSeeker.eu we decide on the following info:

  • Hotel Name, Town, County (State), Star rating, Star accreditor
  • Hotel Photo
  • Room Price from
  • Location

A sample composite image taken from the site is shown below containing the text overlays and location information on a map.


So knowing what we want, how do we go about creating these 7000 images? Options include:

  1. Manual via Paint
  2. 3rd party tool such as Photoshop which allows for a data merge functionality
  3. Create a custom app

Option 1 is clearly a non-starter, but depending on your in house skills options 2 and 3 are the way to go.

For us a custom app seemed the most flexible, as it could pull data direct from the database and integrate future changes via our content management system. So the only question now is what sort of app, options considered were:

  1. Create images via .NET bitmap APIs
  2. Create images via embedded IE web control
  3. Create images via XAML using WPF app

For simple image templates option 1 would be the simplest solution, but we wanted to have the image driven via a content template which pushed us into the direction of either being HTML based or XAML based. In the end we went for XAML as we could then create the templates in Expression Blend and incorporate smart controls such as our hotel map.

So we ended up with a WPF app that pulled the data from the database and then created a custom Bitmap for each hotel based on a XAML template and inject the hotel data. Once we had our custom XAML this was then loaded into a canvas object using the code below:

   1: Canvas oCanvas = (Canvas) XamlReader.Parse(
   2:                 InjectData(xamlTemplate,hotel));
   3: int  iWidth  =  (int)oCanvas.Width;
   4: int  iHeight  =  (int)oCanvas.Height;
   6: oCanvas.Arrange(new  Rect(0,  0,  iWidth,  iHeight));
   8: RenderTargetBitmap  oRenderTargetBitmap  =  new  RenderTargetBitmap(iWidth,  iHeight,  96,  96, PixelFormats.Default);
  10: oRenderTargetBitmap.Render(oCanvas);
  12: // Save image using JpegBitmapEncoder 

Xaml to Bitmap code

So the end result is a direct with 7,000 custom images which can now be used to create the deep zoom collections.

Step 2 - Create deep zoom data

Creating the deep zoom data is the easiest part of this project thanks to the DeepZoom.dll library. A deep zoom collection consists of image tiles and collection meta data.

Using the .NET4 parallel extension the following code will kick off 4 threads to create the deep zoom imagery via the DeepZoom API ImageCreator

   1: Parallel.ForEach(titles, new ParallelOptions { MaxDegreeOfParallelism = 4 }, (title) =>
   2:      {
   3:       new ImageCreator{ MaxLevel = 9 }.Create( imagePath, 
   4:     string.Format(@"{0}\{1}.xml", outputDirectory, title.Id));  
   5:      });
   7: Once the images have been produced we then create the meta data, via the CollectionCreator API:
   9:   new CollectionCreator().Create(titles.Select(t => string.Format(@"{0}\{1}.xml", outputDirectory, t.Id)).ToList(), string.Format(@"{0}\collection.dzc", outputDirectory));

Thanks to blog.smartx.com for this code, used as part of his NetFlix browsing article.

And that’s pretty much it, though using the DeepZoom.dll I couldn’t figure out how to set a minimium zoom level as the tool wanted to create 9 zoom levels, of which levels 0 to 4 are very small and <1k in size, ( need a MinLevel = 3 option)?

Note: Word of warning my 7,000 files (400MB) now went up to 100,000 files (575MB size / 842MB size on disk) and 78,000 folders.

Step 3 of 3 - Create Pivot metadata

In the previous steps we created the deep zoom data, now we need to create Pivot metadata to inform the viewer:

  • How we allow the user to filter or search the information via the filter panel e.g. Number of rooms, Location, price.
  • How we group the results via the sort drop down, e.g. by city, user ratings.
  • What information is displayed in the information panel for a given image e.g. hotel description, contact details etc.
  • Collection Name provides the ability to give the collection a name and (in Silverlight) a custom logo.


For a deep dive on the Pivot meta data see the Collection XML Schema article on Silverlight.net.

Filter panel For us this is still a work in progress, we’ve got the basics such as price, ratings, location, but are still iterating with attention to group of data e.g. hotel features / room features.

Sort Dropdown Care should be taken to not overload the user with irrelevant sort groups, e.g. is group by hotel name really useful? At present we’ve got most of the filter panel appearing in the sort dropdown, but may remove some after testing / user feedback.

Item panel I need to investigate this further, but currently my item panel will only display a button to link to an external URL via the Pivot app, but for the Silverlight control we have to option to multiple buttons and custom event handlers, e.g. to display a Silverlight dialog box. Note: For this release of the Silverlight control you cannot change the size / color of the buttons.

Info Panel Displays data related to the hotel, including extended information pulled from a secondary.cxml data file that holds the hotel descriptions which can be quite large.

The key part of the design here is to decide what information is important to your users. You may want to group things to help the user find the correct info, for example our database has a list of hotel features, but for this panel we broke down the features into groups for hotel and room features to keep the options brief as the panels are quite narrow so text needs to be concise.


  • To reduce the size of downloads I split the collection into 7 collections of ~1000 hotels each. This has the negative effect of searches in the filter panel being restricted to items in the current collection, e.g. if the user is viewing hotels in the South East of England they won’t see any search results for the South West unless they switch collections.
  • Do you want your collection to be consumed by both your Silverlight control, other Silverlight apps, WFP Pivot App? If so then you may want to ensure core functionality does not rely on logic built into your custom Silverlight Pivot Viewer app.
  • The secondary file can be quite large, would be good if that was split into multiple files, not sure if this is possible or not?
  • In the future I may also support dynamic collections e.g. based on search / availability.



This is part 2 of a 3 part series:

Part 1 – What is Pivot

Part 2 – Creating a Pivot Collection (this article)

Part 3 – Creating a custom Silverlight Pivot View application