Replicating Flipboard Part II – Social Signals

Posted by kenshin03 on Feb 28, 2011 in cassius, mac, mobile, tech | 2 comments

Image: http://www.flickr.com/photos/19marksdesign/

A quick update on the project.

Most people never stray far from Flipboard’s default sections (e.g. Tech, Gaming, Fashion) which feature collections of articles from sources hand-picked by the Flipboard editorial team. Pay closer attention though, and you would notice that the laying out of those articles in the Flipboard grids were not simply based on published time or source. Length of the headline, size of the embedded image, similarity of the topics – all these factors seem to come into consideration. Then of course the social strength (tweet count, retweet count, Facebook likes.etc.) of a particular article is one major relevancy factor too. In my opinion, the adoption of a brand new ranking paradigm – social strength, was the single most ground-breaking thinking that was in Flipboard’s design, not unlike how Google invented PageRank and changed the game on search.

This post is not about Flipboard’s layout algorithm (I hope I could do one someday), but rather a quick detour into social signals scraping. The more social signal information we could gather about any article, the easier it would be for us to rank them in terms of interestingness.

After a few hours of fiddling (wasted countless moments not realizing twitter has removed basic authentication support), the two pieces of information I was looking for became accessible.

Twitter Shares

This couldn’t be simpler. Just do a curl to the Count API:

curl "http://urls.api.twitter.com/1/urls/count.json?url=web_site_url"

e.g. curl "http://urls.api.twitter.com/1/urls/count.json?url=http://edition.cnn.com/2011/WORLD/europe/02/10/index.html"

which returns

{"count":4,"url":"http://edition.cnn.com/2011/WORLD/europe/02/10/egypt.protests.london/index.html/"}

Although there’re some limitations. Most notably is how it doesn’t follow redirects and treats different addresses of the same web page as totally separate:

/* all have different results */
curl "http://urls.api.twitter.com/1/urls/count.json?url=http://www.facebook.com/BufSabres/posts/10150099820062954"

curl "http://urls.api.twitter.com/1/urls/count.json?url=http://fb.me/R97GMpug"

curl "http://www.facebook.com/BufSabres/posts/10150099820062954?somefakeparam"

We could definitely use some pre-processing before calling the Count API.

Facebook Likes

Then there’s the beast of Facebook-Likes. Poised as an innocent function for simply declaring your interest on any entity (a web page, a person, an event.etc.), Likes is a tour de force from Facebook’s arsenal for pushing forward their Social Graph vision. When a Like button is added to any web page, that page automatically becomes a living entity in Facebook’s Open Graph repository. In other words, if there’s a Like button added on your web site, Facebook would be able to analyze what type of entity your site represents (a blog, a movie,  a sport team.etc.) and whom expressed interest on it. Now whether that ambition of categorizing the entire web and building social graphs around them is a move forward towards a semantic web or a potentially huge exploit of privacy is for another discussion.

As of now, Facebook’s API does not have a convenient way of returning the Likes count, so as usual, one has to do it via scrapping. The official documentation points to the “Facebook URL Linter” for obtaining the Social Graph info stored by Facebook on any web site. On the URL Linter page, the unique Social Graph ID for a web site would be shown. We could grab that ID and then invoke the Graph API to obtain the Likes count.

e.g. The Matrix page on RottenTomatoes has a Social Graph ID of 119655798047100, and from “http://graph.facebook.com/?id=119655798047100″ we know it’s been liked on Facebook 571 times.

{
   "id": "119655798047100",
   "name": "The Matrix",
   "picture": "http://profile.ak.fbcdn.net/hprofile-ak-snc4/162036_119655798047100_2294421_s.jpg",
   "link": "http://www.rottentomatoes.com/m/matrix/",
   "category": "Movie",
   "website": "http://www.rottentomatoes.com/m/matrix/",
   "description": "Average Rating: 7.4/10 \t\t\tReviews Counted: 126 \t\t\tFresh: 109 | Rotten: 17",
   "likes": 571
}

Except that for reasons unknown, the Linter does not always return the Social Graph ID, like “http://developers.facebook.com/tools/lint/?url=http%3A%2F%2Fwww.oscars.com”, “http://developers.facebook.com/tools/lint/?url=http%3A%2F%2Fwww.apple.com%2F” and many others.

For that matter, perhaps it is perhaps easier for us to bypass the Graph API altogether and directly scrape the iframe that site owners use to add the Like buttons.

All that’s required is a curl followed by some ugly regex filtering code on the returned markup from line 22:

<div id="connect_widget_4d6b574b6fe272202491168" class="connect_widget">
<table class="connect_widget_interactive_area">
<tbody>
<tr>
<td class="connect_widget_vertical_center connect_widget_button_cell">
<div class="connect_button_slider">
<div class="connect_button_container">
<a class="connect_widget_like_button clearfix like_button_no_like">
<span class="liketext">Like</span></a></div>
</div></td>
<td class="connect_widget_vertical_center">
<div class="connect_confirmation_cell connect_confirmation_cell_no_like">
<div class="connect_widget_text_summary connect_text_wrapper">
<span class="connect_widget_facebook_favicon"> </span>
<span class="connect_widget_user_action connect_widget_text hidden_elem">You like <strong>The Matrix</strong>.<span class="unlike_span hidden_elem">
<a class="mls connect_widget_unlike_link">Unlike</a></span>
<span class="connect_widget_admin_span hidden_elem">
<a class="connect_widget_admin_option">Admin Page</a><span class="connect_widget_insights_span hidden_elem">
<a class="connect_widget_insights_link">Insights</a></span></span>
<span class="connect_widget_error_span hidden_elem">
<a class="connect_widget_error_text">Error</a></span></span>
<span class="connect_widget_summary connect_widget_text"><span class="connect_widget_connected_text hidden_elem">You and 611 others like this.</span>
<span class="connect_widget_not_connected_text">611 likes.
<a href="/campaign/landing.php?campaign_id=137675572948107&partner_id&placement=like_button&extra_2=HK" target="_blank">Sign Up</a> to see what your friends like.</span>
<span class="unlike_span hidden_elem">
<a class="mls connect_widget_unlike_link">Unlike</a></span>
<span class="connect_widget_admin_span hidden_elem">
<a class="connect_widget_admin_option">Admin Page</a><span class="connect_widget_insights_span hidden_elem">
<a class="connect_widget_insights_link">Insights</a></span></span>
<span class="connect_widget_error_span hidden_elem">
<a class="connect_widget_error_text">Error</a></span></span></div>
</div></td>
</tr>
</tbody>
</table>
<div id="connect-widget-comment-box-markup">
<!--
<div class="connect_widget_comment_box hidden_elem">
<div class="connect_widget_comment_box_upward_nub"></div>
<div class="connect_widget_comment_area">
<table class="uiGrid" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td><input type="text" class="inputtext connect_widget_comment_textinput DOMControl_placeholder" placeholder="Share it on Facebook with a comment..." value="Share it on Facebook with a comment..." title="Share it on Facebook with a comment..." /></td>
<td><label class="connect_widget_comment_button hidden_elem uiButton uiButtonConfirm" for="u033146_1"><input value="Post" type="submit" id="u033146_1" /></label></td>
</tr>
</tbody>
</table>
</div>
</div>
--></div>
</div>

So here we are, able to extract Twitter Shares and Facebook Likes for any web site. To complete the test, I rounded up a handful of random web sites from Delicious.com and re-ranked them based on social strength. The result was quite clear in terms of showing what kind of web sites might be more appealing to the masses.

Reranking with Social Strengths

Random sites from Delicious, with their social strength scores extracted and normaled

Before and After

A bunch of latest sites from Delicious

Same group of sites, ranked by their Social Strength

Previous:
Part 1: Site Scraping

Ideas scrap pad: Social Empowered Mobile Search

Posted by kenshin03 on Oct 29, 2010 in mobile, tech | 0 comments

A piece of writing from earlier this year (1st April). Still believe there’s a lot of room for innovation in this area and want to keep it at the top of the pile.

screenshot from an early prototype combining Facebook images with Search. Some months later, the Image Search team really did launch such a feature.

Background

As consumer awareness and device capabilities have come of age, the trend in Mobile apps have converged towards hybrids of location based services and social networking. The rise of FourSquare? and Gowalla in the mainstream market only added more fuel to the fire.

Though not obvious at first, this trend has the potential to take Mobile Search to a higher stage of evolution. During the past few years, the uptake of search on mobile devices haven’t been as significant as some might have hailed. While new smartphones come pre-loaded with Search appliances and apps, usage rates compare pale to their desktop offerings.

There’s no denying that in general, user experience of mobile search services leave a lot to be desired, especially when the majority of them were designed to be straight ports of the web search experience. The problem is that the user interface of web search, with typically a text box and a listing of textual results, geared towards offering the user an immersive experience, where he or she is given the illusion of scanning through a long list of related web sites. Due to constraints imposed by hardware, network connectivity, and physical convenience, mobile users rarely find themselves needing to engage in such an intense and immersive search session. It is therefore hardly surprising that keyword based mobile web search has failed to catch on – there just isn’t such a great need.

That’s not to say mobile users lack the intent of searching and discovering information. On the contrary, people on the move are perpetually at the forefront of receiving new data (e.g. running into new people, going to new places, new shops being opened in the neighborhood), and very often find it immensely useful to have additional information at their disposal (say, a guidebook) to help them make sense of the new data. While most people would keep a copy of a local businesses directory in their cars, few consider it an appropriate source for dining advices. When it comes to asking for personal suggestions, nothing beats calling a knowledgeable friend.

Feature Ideas

• Delivery of information streams relevant to a user’s social network AND context (time, location.etc.). For example:
  • Photos uploaded to Facebook by a users’ contact list.
  • New contents and links posted by a users’ contact list in Twitter and Facebook.
  • Search/Filtering functionality provided.
• Supplementary data extracted from social network streams. For example:
  • Which one of my friends have come to this restaurant and when? What were their comments?
  • “Find me a drinking place nearby where Stephanie and Becks would hang out, but not Vicky?”
  • Any special unlocked Foursquare venues nearby?
• Forms of input
  • De-emphasize the classic search box and listings UI. Use location, direction, time, data mining.etc. to direct the search experience.

Resources

• FourSquare API
• Twitter API
• TweetPhoto API
• Facebook Connect

Yahoo’s Facebook Album Search:

• Explore Interesting, Personal Photos on Yahoo! Search

A little goes a long way: NSOperations

Posted by kenshin03 on Mar 23, 2010 in mac, mobile, tech | 0 comments

While putting together a prototype that pulls a feed of interesting pics from flickr and have them displayed as a carousel on a UIScrollView, I was baffled by the unusual length of time it took for the scrollview to render, even in the simulator. The delay, especially when combined with the use of a UINavigationController, was downright unacceptable.

Check out this vid to get a taste:

It took only one look from a veteran objective-c developer colleague to diagnose the problem – UINavigationController was waiting for a single thread to finish rendering all 60 images before sliding in the ScrollView. “NSInvocationOperation is your answer“, he said.

And indeed it was, the perceived performance sky rocketed ten folds after the process of initializing UIImages was wrapped in an NSOperation. Here’s the difference:

And the code change was as follows:

Basically, all that was required was to move the necessary rendering logic to a separate method, create instances of NSOperations with selectors set to the method, and add those NSOperations into the NSOperationQueue. The Queue would schedule the firing of these operations and there wouldn’t be any hint of sluggishness at the UI. Thread programming 101. Duh.

Extended Reading:

• http://developer.apple.com/cocoa/managingconcurrency.html

My thoughts on EFF’s “All Your Apps Are Belong To Apple” rant

Posted by kenshin03 on Mar 16, 2010 in mac, mobile, tech | 0 comments

WTEFF?

So I was recently asked of my opinion on EFF’s “All Your Apps Are Belong To Apple” posting which has sparked some discussions for a day or two.

My reactions? Hardly surprised, really. In fact, although the arguments presented appear reasonable, the article has a serious air of deja-vu to it that provokes more doldrum rather than rage upon Apple followers.  Every few months, the old statement of “Apple treating its customers unfairly” would get recycled time and again. A few such litigation came to mind immediately:

• “Apple should open up the iPod for sync-ing with other music formats.”
• “Apple should allow songs purchased from iTunes store to be playable on other mp3 players.”
• “Apple should allow Mac OS X to be installed on regular PCs.”
• “Apple should allow consumers to jailbreak the iPhone because they paid for the device and could do anything to it as they pleased.”
• “Apple should allow people to replace the batteries on their Macbooks/iPhones.”

Funny thing is, these never amount to anything substantial (like a court order for Apple to change its policies). Pundits would complain as much as they could, but every argument raised against the company would almost certainly be silenced by a larger group of supporters in the end. Some dismiss this as mere fanboyism, fused by leagues of blindly loyal geeks prepared to accept any offerings or services that came out of 1st Infinite Loop without ever questioning the fairness and sanity of it all.

When you think about it, this explanation doesn’t really hold. With more than 220 million iPods, 57m iPhones, 12m Macs (2009) sold, the installation base is huge and reaches much further than tech-heads and nerds. Is the majority of people in this large user base simply oblivious to the various restrictions imposed by Cupertino then? Why are people willing to pay a premium in exchange of Apple services or products that are apparently ridded with limitations? Flashy marketing and peer pressure alone doing the magic? Obviously not.

At the end of the day, it’s really down to the outstanding user experience offered by Apple products and services in general. Ever since 1977, when Steve Jobs decided he would hand-polish the plastic cases of Apple ][ machines with sand paper to "make the corners smoother to the touch", the company has been embarking on a never-ending mission of striving to produce "unparalleled user experiences" in everything they do or make. Even when that meant going against the entire computer industry and alienating their loyal user base by imposing strict EULAs or removing features that were popular amongst users.

Cases in point - getting rid of floppy drives in iMacs and 1394 ports on iPods/Macs, killing the iPod mini, purging more than 5000 mild-sex oriented apps from AppStore overnight, suing third-party accessories makers, enforcing non-removable batteries on Macbooks and iPhones, forgoing physical keyboards on iPhones…in other words, imposing a water-tight system and employing draconian measures that fiercely go against anyone or anything daring to stand in its path of evolution.

Openness comes with a price.

Each of these steps created uproars and distress upon introduction, but when the dust has settled, favorable opinions would come sifting through. The controversial change brought about by Apple would become a de-facto standard, and pretty soon, competitors who ridiculed the change initially would begin to jump on the bandwagon and roll out inferior copycat products/services to cash in.

So…back to the point. Does this 30 year old track record render the points EFF highlighted in the iPhone Development Program License Agreement justifiable? That remains to be seen, but knowing that control-freak motif behind the company might make those restrictions more reasonable:

• App Store Only: this is the only way Apple can make sure that apps available for the iPhone are reviewed and up to a certain standard, and could be revoked anytime if need be.
• Ban on Reverse Engineering: duh. This is common sense right? Apple develops and owns the iPhone OS and its core apps, and need to protect its investment one way or another.
• No Tinkering with Any Apple Products: same as the App-Store-only sentiment.
• Kill Your App Any Time: this is the only way to ensure crap apps like I’m Rich and porn/malicious apps can be removed at anytime.

Consumers associate everything they see on the iPhone with the Apple brand, so it is of no surprise that the company does everything to protect its decades-old reputation.

Having said that, there always follow the question of whether Apple Inc is “right”, because many believe that the companies in the industry should play a part in “fostering innovation” by “promoting openness and interoperability”. After all, M$ got her share of negative opinions by being a monopolistic goliath that stifle innovation in return for cash. Shouldn’t Apple Inc be subject to the same level of scrutiny?

No matter how rosy you paint a picture, it's no use if we'll still stuck on the other side of the wall.

In my opinion, that can’t be answered by anyone until the legacy of Apple Inc, Microsoft, Google, Yahoo and all other “monopolistic” companies are history and the market has entered a new phase. Consider J2ME (arguably the predecessor of Android) – a standard that was backed by all major software/hardware companies, was totally open but yet was also totally chaotic and totally failed to be grasped by consumers at all. Yes, the openness of J2ME was great for the tech crowd and manufacturers. But the side effect was fragmentation and incoherence. At the end of the day,  openness became the very thing that killed the market it wanted to create in the first place, and when the market is gone, so are your promises for innovation. All efforts became futile.

I think the day we should start worrying is the day when it comes to a point Apple Inc became the only choice available, much like how M$ Office and IE stipulated the markets back then. Until then, as a consumer and developer, I’d still be thrilled by what the iPhone platform has to offer.

p.s. Come to think of it, I should have answered this question simply with “Hey, the success of the AppStore meant that I could now throw Hadokens against Blanka on the bus!” Now that’s an argument.

Extended Reading:

• http://www.businessweek.com/the_thread/techbeat/archives/2008/03/apples_design_p.html – Apple’s design process
• http://www.businessinsider.com/the-case-against-apple-2009-8
• http://techcrunch.com/2009/07/27/apple-yanks-the-cord-on-gv-mobile-is-it-trying-to-kill-google-voice-on-the-iphone/
• http://www.wired.com/gadgets/wireless/magazine/16-02/ff_iphone?currentPage=all – “The Untold Story: How the iPhone Blew Up the Wireless Industry”
• http://www.rollingstone.com/news/story/5939600/steve_jobs_the_rolling_stone_interview
• http://thenextweb.com/mobile/2010/03/15/android-market-breaks-250-million-downloads-989-free-apps/

iPhone OS app dev – the basics (again!)

Posted by kenshin03 on Mar 11, 2010 in mac, mobile, tech | 0 comments

Re-done properly with self-made code samples! Strictly for newbies.

Developing native iPhone apps can be fun, as SDK 3.0 offers tons of APIs and options that makes leveraging of features like location detection, animation and embedded maps in your apps effortless.

Then again, developing native iPhone apps can be a major pain, as the restrictions imposed by the SDK can be so strict that you need to go to lengths to create something simple.

This sharing aims to quickly cover the basics (iphone OS, objective-c, UIKit elements, ViewControllers). You’ll also be able to see what it takes to add features like Animation, Network Connectivity, JSON parsing and iPhone specific features such as Shake/Orientation Change/Gestures Detection to an app.

Admob-ing “Chinese Girls” by code

Posted by kenshin03 on Nov 29, 2009 in mac, mobile, tech | 0 comments

Upon seeing the first admob ads from Paper Toss, the realization that mobile display ads have entered a new era hit home. The catch-phrases of yesterday like m-coupons and location based advertising have given way to new ecosystems of ad networks and interconnectivity. This post is a brief account of my first foray into enabling admob in an iPhone app, using the source code to “Chinese Girls”.


First of all, anyone should check out Admob’s developer site and create a free account. Before downloading the sample code one is required to fill in details about the app where ads would be served. I don’t think Admob actually check if the are authentic though, as there’s no way they can verify who the owner of the app is.

Having done all the form filling, a link to download the SDK would be given. Note that the developer key is dynamically embedded in the sample code itself, which means it would be a bad idea to take a binary passed on by a friend and use it directly without even changing the developer key in the source code.

The SDK contains some sample code and library files an app serving admob ads need to include. Admob code makes use of the TouchJSON framework so the binaries to that are given too.

From then on it’s simply following instructions in the README file for the integration. Now’s the time to open up the project file to “Chinese Girls” with Xcode.

The first step would be adding the necessary library code and headers to the project. Since Chinese Girls is a 3.0+ app, the couple of .a files have to be renamed (simply removing the “3_0″ suffix) and grabbed from the extras/folder instead:

The next step is simply adding required frameworks. Admob documentation explains that these some framework files are needed to be there for successful compilation only, and they should not add to the weight of the app.

Now comes the real integration. Suppose I want to have the ad showing at the bottom of the main menu page, I would need to make the ViewController class for that page an AdMob delegate.

Changes highlighted in MenuViewController.h:

Finally, some boiler plate delegate code would need to be added to the MenuViewController.m class itself. For testing, return YES in the useTestAd() function. These delegate methods were lifted directly from samples in the Admob SDK. Except some customizations were needed in the didReceiveAd method. To position the ad at the bottom of the page, the frame holding the ad was appended as a subview of mainView.

Note that ad Admob app don’t really care about the existing visual elements, so as we could see, the info icon at the bottom of the screen got blocked and became inaccessible. Move it up 40px and a re-compilation makes it see the light of day again. However, I can already foresee that more complicated re-positioning logic would be required in the didReceiveAd() method when the screen composes of more elements.

So that wasn’t too hard. In the next post I shall be taking interstitial ads (video ads that appear between view transitions) for a spin and see whether that would call for greater integration efforts.