While reading Revolution in the Valley some years back, one of the things that moved me was how Apple evangelists would voluntarily go off to electronics stores and offer to “fix” the retail Apple experience – ensuring software was running properly, answering questions from customers and sales reps, and the like. They were the first generation Apple Geniuses.

At one time or another, each Mac user would undoubtedly come across the chance to do his bit of evangelizing. Mine was about three or four years ago, when our department was still flat-out filled with XP laptops (with the anti-virus kicking in and commanding 99% CPU hourly) and only a handful of folks owned Apple machines. Refused to be issued a Macbook by IT, I sought to bring in my own instead. That sparked a bit of interest among my team mates (and grunt from management), especially those planning to replace their notebooks.

After hearing remarks like how the “Mac OS looks like Windows”, I had an urge to set the record straight and tell the team about the story of Apple and Steve Jobs. I consider it essential knowledge for anyone who works in the field.

Here’re the slides I showed them that afternoon. A fitting end for an unforgettable week I guess.

Photo courtesy of turtlemom_nancy @flickr

Like many others I was shocked by the news of Steve Job’s passing away when I woke up this morning. Sure, the writing has been on the wall for quite some time and his publicized resignation more or less paved way for what is to come, but that doesn’t make the news any easier to swallow when it strikes.

The world as we know it will never be the same. For many years to come, people will still be discovering how much they have truly lost, and occasionally pondering how Steve would have done things differently if he is still around.

Most people weigh upon his contributions by the products he helped to create, his innovative visions or business acumens. Those are impressive, but I’m more touched by the more human side of the man. His relentlessness in pursuing true greatness. The drive to create works of wonder that are in harmony with nature, people, the world. His refusal to be let down by the status quo. His eagerness to shine among the flock. And the sense of proudness and satisfaction when he sees people appreciating his work. To that end, this mentality is no different than that of a fourth grader at school striving hard to win the praises of parents and teachers.

Some people singled that out as a fault in Jobs’ character. That his biological parents put him up for adoption created a vacuum in his heart he spent a lifetime trying to reconcile. In my opinion, that aspect was a gift rather than a fault. It was probably one of the greatest source of strength in Steve’s life, and as a result the single most important aspiring quality contributing to the high levels of standard carried forward to most of Apple’s products.

If only could more people in the world retain this child-like innocence and act upon it in everything they do, the human race would benefit from more amazing breakthroughs than an array of new laptops and mobile phones every year. That, would be the greatest legacy that Steve Jobs could have left behind.

steve jobs 1955-2011

Once crossed the line into the mobile word, however, these three factors lose their usefulness drastically. Text input on mobile devices is largely impractical and traditional web pages don’t render well, so media discovery and consumption on mobile devices is generally inferior compared to the same experience from printed mediums like newspapers and magazines. While big name players attempt to tackle the issue simply by snapping on extra features (Google Mobile Voice Search, Google Instant Preview for Mobile.etc.), the underlying problems remain resolved as its ranking algorithm is the same as its desktop counterpart. Flipboard is so great because, in my opinion, it has found and defined the new three pillars of relevancy for mobile content consumption and they are freshness, social, and readability* – and they work wonderfully.

With this in mind, I put some work to the server components of Cassius. From a simple script that turns a Tweet into a JSON feed, the pipeline now includes saving documents into a transitional store (MongoDB) and a series of quality measurement calculations. While the extra processing means we won’t be able to serve the feed in realtime, the cost should be worthwhile and I hope the results justify that.

How well does your article read?

In Zite or Flipboard, it’s not uncommon to run into articles with summary texts that resemble gibberish (see below). The issue is often a result of incorrect identification of raw HTML elements as meaningful content, and is very hard to avoid. I have seen attempts to solve this problem using NLP and machine learning classification methods, to varying degrees of success. Since those are beyond my capabilities, I opted to use some traditional methods to measure the quality of a piece of writing – by taking its readability metrics. From Wikipedia, readability evaluation refers to “the ease in which text can be read and understood“, and “…various factors to measure readability have been used, such as speed of perception, perceptibility at a distance, perceptibility in peripheral vision, visibility, the reflex blink technique, rate of work, eye movements, and fatigue in reading…“.  Readability metrics measurement tools are widely available, and embedded in word processors and email clients.

source: corporategeek.info

results of bad scraping

In a nutshell, the tools apply different statistical formulas on a piece of English writing, and the resulting scores form an impression of its understandability. The formulas typically break text into syntactic components such as words and sentences and count their distribution or frequency in relation to the text being analyzed. The most common readability formulas and descriptions are given below:

I found it more pleasing to read blog posts and articles on Flipboard/Zite that are about a page in length. Contents that span multiple pages are too demanding for casual reads, while short tweets or one liners aren’t worth the two clicks effort to expand and shrink them from the page (yes really). For simplicity, let’s take my reading habits as standard, and use the following thresholds for computation:

• Flesch – 50 (Times magazine has a score of about 52)
• Flesch-Kincaid – 13 (pre-college level)
• Gunning Fog – 12 (texts for wide audience have fog index of less than 12)
• SMOG - 13 (pre-college level)
• Coleman Liau - 13 (pre-college level)
• ARI - 13 (pre-college level)

In the next post we’ll continue to explore the three pillars, and look at some test results to see whether the additional aspect of readability would help us create a feed that is better optimized for the user’s final reading experience.

References on the topic is plentiful, but surprisingly the most useful one I came across was a paper published in 1977 titled “Computer Assisted Layout of Newspapers” by the MIT. You can find the full 184 pages here. The paper is a gem to read and goes into detail on even how ads and pictures layouts could be automatically assigned to a theoretical newspaper page. I shall definitely return to it for more inspiration, but so far I have based the design of the prototype on Chapter 6, A Symbolic Graphics Language For News Layout.

The diagram below lifted from P.84 of the paper tells it all. Pages on Flipboard largely employ a rows/columns layout combination, and the powerful template language described in the paper should be able to cover all variations effortlessly .

a simple yet powerful layout language

Note that I cheated a little and defined my version of the template language in JSON, mainly for easier parsing in Objective-C.

Therefore,
P1 || (S1 = S2) is represented with {"columns": [{"type":"P1"}, {"rows":[{"type":"S1"}, {"type":"S2"}]}]} in my app,

and

S3 || (S4=(S5 || S6 || S7)) becomes {"columns":[{"type":"S3"}, { "rows": [{"type":"S3"}, {"columns":[{"type":"S5"},{"type":"S5"}, {"type":"S7"}]}] }]}.

With a structure like this, we could simply parse the JSON into multi-dimensional arrays (e.g. {“P1″, “{S1, S2}”}), then write classes to traverse the array and return suitable UIViews or collections of UIViews. Only two-level nesting is supported in the code right now.

The UIView generation process itself is just as crude at present. While looping through the array, the type of value stored is examined, and if it’s a definition like “P1″ or “TIA”, a helper class would create the corresponding UIView, with arguments being the article itself and attributes like the size of the array passed in for presentation purposes. All these take place in the PageLayoutManager class. A whole lot more work will be put in around these classes.

I’m hoping that more help from the server-side will be used for both the templates definition and articles selection process. Analysis on word count, images in the article, source authority, social signals and other relevancy factors should already been taken into account by the time these articles and templates arrive at the client app.

Finally, here’s the template used for generating the pages shown in the first video. There are four pages altogether, with pages 1 and 4 being row-based and pages 2 and 3 column-based. These layout designs are quite similar to the ones used heavily on tweets-display pages on Flipboard.

The rendering:

page 1 - row 1 is article, row 2 three columns of articles.
page 2 - column 1 is article, column 2 is 2 rows of articles.
page 3 - 1 column, 1 article.
page 4 - 2 rows of articles

Remote or Local?

A colleague pointed out the template definitions must be defined and stored on the client app locally, as the app shouldn’t need to fetch a new template from the server when the device changes orientation. I haven’t thought about that yet. To me, it makes more sense to have the server picking templates that are more suited to the content being served. I’m totally not thinking about how to deal with landscape orientations yet.

Extended Reading:

• COMPUTER-ASSISTED LAYOUT OF NEWSPAPERS. Reubtures et al.
• Optimization of web newspaper layout in real time. J. Gonzalez et al. / Computer Networks 36
  ]]> http://www.corgitoergosum.net/2011/06/22/knitting-a-page/feed/ 0 http://www.corgitoergosum.net/2011/06/16/cassius-is-on-github/ http://www.corgitoergosum.net/2011/06/16/cassius-is-on-github/#comments Thu, 16 Jun 2011 10:43:16 +0000 kenshin03 http://www.corgitoergosum.net/?p=917 Finally got round to putting together a decent enough client! Although the code is pretty rough now and the app would crash after a while, but at least it’s a start innit?!

  Repos:

  https://github.com/kenshin03/camus (server side, java)
  https://github.com/kenshin03/cassius (iOS client side)

  The article pages were generated by a custom template that’s defined in JSON, and the image on the cover page is grabbed from Instagram’s API:

  Cassius 20110616

  View more videos from kenshin03

  The iconic page flip effects were lifted straight out from AFKPageFlipper. Thanks again Marco!

  ]]> http://www.corgitoergosum.net/2011/06/16/cassius-is-on-github/feed/ 9 http://www.corgitoergosum.net/2011/06/12/replicating-flipboard-part-iv-prelude/ http://www.corgitoergosum.net/2011/06/12/replicating-flipboard-part-iv-prelude/#comments Sat, 11 Jun 2011 17:13:41 +0000 kenshin03 http://www.corgitoergosum.net/?p=845 It’s been forever since the last update. But amidst a trillion other things, this Flipboard study never strayed far from my mind (and the IDE).

  More detailed posts on the progress and designs of the prototype will come this week, but right now, I can’t wait to share the very first screenshots of Cloneboard Cassius.

  What you see here is a cover page using a random image from Instagram transitioning into the first articles page showing stories from my Twitter feed. The layout was generated dynamically from a custom template defined in json.

  

  I’ll be putting all the code up on Github for better back up and version control soon.

  ]]> http://www.corgitoergosum.net/2011/06/12/replicating-flipboard-part-iv-prelude/feed/ 4 http://www.corgitoergosum.net/2011/03/06/replicating-flipboard-part-iii-%e2%80%93-how-flipboard-lays-out-content/ http://www.corgitoergosum.net/2011/03/06/replicating-flipboard-part-iii-%e2%80%93-how-flipboard-lays-out-content/#comments Sun, 06 Mar 2011 11:47:18 +0000 kenshin03 http://www.corgitoergosum.net/?p=754 Shifting the focus back to the iPad app, let’s take a look at how Flipboard processes and lays out Facebook and Twitter feeds. Does it really employ any social signal based ranking?

  Here’s a sample of some Twitter feeds I receive, shown in a browser and in Flipboard:

  Sample Twitter Feed

  Same group of Twitter feeds viewed in Flipboard

  Alright…what can be deduced from this? A few things became obvious when the same data is shown side-by-side:

  • Ranking of the Twitter articles in Flipboard more or less retains the original sort-by-time order! Don’t think there’re any clever hidden social ranking at play here.
  • Some Tweets were dropped by Flipboard, presumably because they do not contain links. Examples like (#3, #11, #15, #17, #18) are simply Twitter conversations.
  • Tweets that have links to sites with Images (e.g. #7, #9) seemed to be given higher display priority.

  What about Facebook? The typical Facebook feed is a lot more complex, with mixed content ranging from Check-ins, uploaded lmages, to Likes and notifications from whatever apps/games a user has added. The prospect of having to analyze so many different types of content itself is daunting enough, let alone the additional efforts of re-ranking and laying them out in an app.

  As above, this is a snapshot of my Facebook News feed (names, faces and updates blurred out – don’t worry people!) in a browser and in Flipboard:

  facebook news feed

  facebook in Flipboard

  Immediately we are striked with much more intriguing findings:

  • Flipboard completely ditched the published time of the feed articles and laid them out entirely based on readability attributes (text length, image size.etc.).
  • Quite a few Facebook articles were dropped:
    • Article #5 – a Facebook Places-Checkin: Due to lack of images?
    • Article #6 – An Image uploaded from iPhone. Not sure why it wasn’t shown in Flipboard.
    • Articles #7, #13 – These were “xxx started using xxx app” messages, which contain no links or images and frankly, nothing interesting.
    • Articles #20, #21 – These were messages from Groupon. As article #14 was also a Groupon article, I guess #20 and #21 were dropped as Flipboard prevents showing too many articles from the same source all at once.

  And while looking at Facebook articles, we might as well take a quick look again if Facebook-Likes contribute to the layout or ranking at all. The verdict – NOPE. If social signals play a major role in ranking, then #12 and #19 should have higher placements in Page 1 or 2 at least.

  • #1 – 1 reply
  • #2 – 1 Like
  • #3 – 0 Likes (but 82 youtube Likes)
  • #8 – 0 Likes
  • #4, #9 – 0 Likes
  • #11 – 4 Likes, 7 Comments
  • #12 – 1 Reply, 92 Likes
  • #14 – 0 Likes
  • #16 – 3 Comments
  • #19 – 34 Likes
  • #17 – 4 Likes
  • #18 – 2 Likes
  • #22 – 2 Likes

  From such observations, I suspect Flipboard conducts workflows that roughly compose of the steps below:

  Likely processing workflow for Facebook feeds

  Likely processing workflow for Twitter feeds

  Initially, I found it hard to accept this interpretation of templates-over-content. It is basically saying there’s no magic, that Flipboard merely puts together a random collection of page templates then proceeds to filling those templates with the next most suitable article from a content feed.

  Putting an end to this suspicion, I switched on Flipboard after a few more articles emerged in my Facebook feed. Given a largely unchanged set of data, if Flipboard employs a content-centric ranking criteria, then the layout of the pages should remain more or less the same? From the screenshots below, this was clearly not the case. Notice how the same data was laid out much more differently:

  Original render in Flipboard

  Second render of the same Facebook feed

  So that concludes the brief study on Flipboard’s layout algorithm. From this point on I shall start firing up the IDEs and getting my hands dirty on building that prototype. Pretty thrilled by the amount of attention and support this little project seems to be garnering – thanks and please stay tuned!

  Previous Post
  
  
  
  Related:

  • how are the username and password used/stored for Google Reader?
  ]]> http://www.corgitoergosum.net/2011/03/06/replicating-flipboard-part-iii-%e2%80%93-how-flipboard-lays-out-content/feed/ 5 http://www.corgitoergosum.net/2011/02/28/replicating-flipboard-part-ii-social-signals/ http://www.corgitoergosum.net/2011/02/28/replicating-flipboard-part-ii-social-signals/#comments Mon, 28 Feb 2011 10:32:22 +0000 kenshin03 http://www.corgitoergosum.net/?p=725

  Image: http://www.flickr.com/photos/19marksdesign/

  A quick update on the project.

  Most people never stray far from Flipboard’s default sections (e.g. Tech, Gaming, Fashion) which feature collections of articles from sources hand-picked by the Flipboard editorial team. Pay closer attention though, and you would notice that the laying out of those articles in the Flipboard grids were not simply based on published time or source. Length of the headline, size of the embedded image, similarity of the topics – all these factors seem to come into consideration. Then of course the social strength (tweet count, retweet count, Facebook likes.etc.) of a particular article is one major relevancy factor too. In my opinion, the adoption of a brand new ranking paradigm – social strength, was the single most ground-breaking thinking that was in Flipboard’s design, not unlike how Google invented PageRank and changed the game on search.

  This post is not about Flipboard’s layout algorithm (I hope I could do one someday), but rather a quick detour into social signals scraping. The more social signal information we could gather about any article, the easier it would be for us to rank them in terms of interestingness.

  After a few hours of fiddling (wasted countless moments not realizing twitter has removed basic authentication support), the two pieces of information I was looking for became accessible.

  Twitter Shares

  This couldn’t be simpler. Just do a curl to the Count API:

  curl "http://urls.api.twitter.com/1/urls/count.json?url=web_site_url"

  e.g. curl "http://urls.api.twitter.com/1/urls/count.json?url=http://edition.cnn.com/2011/WORLD/europe/02/10/index.html"

  which returns

  {"count":4,"url":"http://edition.cnn.com/2011/WORLD/europe/02/10/egypt.protests.london/index.html/"}

  Although there’re some limitations. Most notably is how it doesn’t follow redirects and treats different addresses of the same web page as totally separate:

  /* all have different results */
  curl "http://urls.api.twitter.com/1/urls/count.json?url=http://www.facebook.com/BufSabres/posts/10150099820062954"
  
  curl "http://urls.api.twitter.com/1/urls/count.json?url=http://fb.me/R97GMpug"
  
  curl "http://www.facebook.com/BufSabres/posts/10150099820062954?somefakeparam"

  We could definitely use some pre-processing before calling the Count API.

  Facebook Likes

  Then there’s the beast of Facebook-Likes. Poised as an innocent function for simply declaring your interest on any entity (a web page, a person, an event.etc.), Likes is a tour de force from Facebook’s arsenal for pushing forward their Social Graph vision. When a Like button is added to any web page, that page automatically becomes a living entity in Facebook’s Open Graph repository. In other words, if there’s a Like button added on your web site, Facebook would be able to analyze what type of entity your site represents (a blog, a movie,  a sport team.etc.) and whom expressed interest on it. Now whether that ambition of categorizing the entire web and building social graphs around them is a move forward towards a semantic web or a potentially huge exploit of privacy is for another discussion.

  As of now, Facebook’s API does not have a convenient way of returning the Likes count, so as usual, one has to do it via scrapping. The official documentation points to the “Facebook URL Linter” for obtaining the Social Graph info stored by Facebook on any web site. On the URL Linter page, the unique Social Graph ID for a web site would be shown. We could grab that ID and then invoke the Graph API to obtain the Likes count.

  e.g. The Matrix page on RottenTomatoes has a Social Graph ID of 119655798047100, and from “http://graph.facebook.com/?id=119655798047100″ we know it’s been liked on Facebook 571 times.

  {
     "id": "119655798047100",
     "name": "The Matrix",
     "picture": "http://profile.ak.fbcdn.net/hprofile-ak-snc4/162036_119655798047100_2294421_s.jpg",
     "link": "http://www.rottentomatoes.com/m/matrix/",
     "category": "Movie",
     "website": "http://www.rottentomatoes.com/m/matrix/",
     "description": "Average Rating: 7.4/10 \t\t\tReviews Counted: 126 \t\t\tFresh: 109 | Rotten: 17",
     "likes": 571
  }

  Except that for reasons unknown, the Linter does not always return the Social Graph ID, like “http://developers.facebook.com/tools/lint/?url=http%3A%2F%2Fwww.oscars.com”, “http://developers.facebook.com/tools/lint/?url=http%3A%2F%2Fwww.apple.com%2F” and many others.

  For that matter, perhaps it is perhaps easier for us to bypass the Graph API altogether and directly scrape the iframe that site owners use to add the Like buttons.

  All that’s required is a curl followed by some ugly regex filtering code on the returned markup from line 22:

  <div id="connect_widget_4d6b574b6fe272202491168" class="connect_widget">
  <table class="connect_widget_interactive_area">
  <tbody>
  <tr>
  <td class="connect_widget_vertical_center connect_widget_button_cell">
  <div class="connect_button_slider">
  <div class="connect_button_container">
  <a class="connect_widget_like_button clearfix like_button_no_like">
  <span class="liketext">Like</span></a></div>
  </div></td>
  <td class="connect_widget_vertical_center">
  <div class="connect_confirmation_cell connect_confirmation_cell_no_like">
  <div class="connect_widget_text_summary connect_text_wrapper">
  <span class="connect_widget_facebook_favicon"> </span>
  <span class="connect_widget_user_action connect_widget_text hidden_elem">You like <strong>The Matrix</strong>.<span class="unlike_span hidden_elem">
  <a class="mls connect_widget_unlike_link">Unlike</a></span>
  <span class="connect_widget_admin_span hidden_elem">
  <a class="connect_widget_admin_option">Admin Page</a><span class="connect_widget_insights_span hidden_elem">
  <a class="connect_widget_insights_link">Insights</a></span></span>
  <span class="connect_widget_error_span hidden_elem">
  <a class="connect_widget_error_text">Error</a></span></span>
  <span class="connect_widget_summary connect_widget_text"><span class="connect_widget_connected_text hidden_elem">You and 611 others like this.</span>
  <span class="connect_widget_not_connected_text">611 likes.
  <a href="/campaign/landing.php?campaign_id=137675572948107&partner_id&placement=like_button&extra_2=HK" target="_blank">Sign Up</a> to see what your friends like.</span>
  <span class="unlike_span hidden_elem">
  <a class="mls connect_widget_unlike_link">Unlike</a></span>
  <span class="connect_widget_admin_span hidden_elem">
  <a class="connect_widget_admin_option">Admin Page</a><span class="connect_widget_insights_span hidden_elem">
  <a class="connect_widget_insights_link">Insights</a></span></span>
  <span class="connect_widget_error_span hidden_elem">
  <a class="connect_widget_error_text">Error</a></span></span></div>
  </div></td>
  </tr>
  </tbody>
  </table>
  <div id="connect-widget-comment-box-markup">
  <!--
  <div class="connect_widget_comment_box hidden_elem">
  <div class="connect_widget_comment_box_upward_nub"></div>
  <div class="connect_widget_comment_area">
  <table class="uiGrid" cellspacing="0" cellpadding="0">
  <tbody>
  <tr>
  <td><input type="text" class="inputtext connect_widget_comment_textinput DOMControl_placeholder" placeholder="Share it on Facebook with a comment..." value="Share it on Facebook with a comment..." title="Share it on Facebook with a comment..." /></td>
  <td><label class="connect_widget_comment_button hidden_elem uiButton uiButtonConfirm" for="u033146_1"><input value="Post" type="submit" id="u033146_1" /></label></td>
  </tr>
  </tbody>
  </table>
  </div>
  </div>
  --></div>
  </div>
  

  So here we are, able to extract Twitter Shares and Facebook Likes for any web site. To complete the test, I rounded up a handful of random web sites from Delicious.com and re-ranked them based on social strength. The result was quite clear in terms of showing what kind of web sites might be more appealing to the masses.

  Reranking with Social Strengths

  

  Random sites from Delicious, with their social strength scores extracted and normaled

  Before and After

  

  A bunch of latest sites from Delicious

  

  Same group of sites, ranked by their Social Strength

  Previous:
  Part 1: Site Scraping

  ]]> http://www.corgitoergosum.net/2011/02/28/replicating-flipboard-part-ii-social-signals/feed/ 2 http://www.corgitoergosum.net/2011/01/17/replicating-flipboard-part-i-site-scraping/ http://www.corgitoergosum.net/2011/01/17/replicating-flipboard-part-i-site-scraping/#comments Mon, 17 Jan 2011 13:14:50 +0000 kenshin03 http://www.corgitoergosum.net/?p=673 Having taken a long good look at the social magazine Flipboard, it was time to dig beneath the cover and contemplate what kind of technology lies behind its minimalistic interface.

  I began by studying the inconspicuous yet critically important Reader feature. Personally, if Flipboard was to pop open mobile Safari each time I clicked into an article, I would have deleted the app right away, so that to me is the number 1 major feature. One of the wonders of Flipboard was how it manages to scrape relevant content from web sites in such an elegant and effortless manner.

  

  Expanded web content view on Flipboard. wpcdn.padgadget.com

  Part of my job these years has been to do with web scrapping and crawling, and all the more because I know how scrappers and crawlers work, the speed and precision that Flipboard does it left me very impressed – there are apparently no machine learning models, no editorial selection, no reliance on meta data, no nonsense. Most of the time, you just throw any links at  the app – Facebook, bitty, flickr, youtube vids, RSS, and It swallows it and churns out their core contents in the most aesthetically pleasing format like a humble neo-classism painter. What technical options out there are capable of performing such an achievement?

  

  Safari Reader

  After days of testing and failing, I was prepared to throw in the towel, abandon the idea and go back to reading about iOS and cocos2d – until incidentally a little blue icon on the desktop screamed out at me. That was the “Reader” icon in Safari’s toolbar.

  Introduced in Safari 5, the Reader is a feature that extracted the most prominent contents of a web page and displays them in a clutter free overlay. The Reader works best with pages that are text heavy, such as news articles and the like. Now that’s not unlike what Flipboard does.

  

  Content extracted by Safari Reader

  Is there any chance that Flipboard employs a similar scraping technique as Apple? Would it take us a step closer if somehow we could crack Safari Reader’s secret? Clues were revealing themselves here and there. After a bit of googling, I landed on the site of the Readability project by a group called Arc90.

  The main works of the Readability project is a JavaScript file that parses the dom tree of a document and extracts the section where the most relevant content lies. I threw a few URLs at Readability’s test page, and was pleasantly surprised  with how comparable the results were with Safari’s Reader, even for more complicated pages (some sources say that Safari does ship with a version of Readability.js bundled in).

  Since the JS isn’t very viable for running in non-browser environments, I reverted to the php port by Keyvan Minoukadeh. With some extra cleaning logic thrown on top, I was able to get very good results with the sample news article pages.

  Source: Times article versus  readability php extracted result

  

  original article on Times.com

  

  Left to right: Flipboard render, Safari 5 Reader, Readability php port

  Source: A cartoon on cracked.com versus  cracked.com article scrapped by readability php

  

  A cartoon page on cracked.com

  

  Clockwise: Safari 5 wouldn't show "Reader" mode, Readability php extracted gibberish JS code, Flipboard gave up crawling page; shows url instead

  Surely the tool was not without limitations. With looser or pages that aren’t text centric, the results are often gibberish and unusable. For instance, the Readability would throw its hands up in the air when given a Flickr photoset URL.

  Flipboard most probably employs several scrapping techniques suited to different types of  web pages. With lots more to study, I shall leave the scraping part at that and revisit this space later.

  Continued:
  Part II – Social Signals

   

  Scraping and other issues:

  • How @flipboard does image cropping. Interview of CTO @avh
  • Is Flipboard legal
  • From a lawyer’s point of view
  • About Flipboard and reading surfaces

  

  ]]> http://www.corgitoergosum.net/2011/01/17/replicating-flipboard-part-i-site-scraping/feed/ 7