OMGPOP Development

Now open sourced: Avery.

Fri, 16 Dec 2011 12:40:00 -0500

Avery is a real-time time-based metric storage, retrieval, and graphing service. It is entirely javascript.

At OMGPOP we use Avery to see real-time, and historical, metrics across all of our games, applications and systems to help predict trending for resource(servers/people) allocation. When needed, Avery also allows us to get a high-resolution view of certain metrics from only a few parameters.

Example:

You can find out more about Avery over on GitHub: https://github.com/OMGPOP/avery

Game Hack Day - NYC

Fri, 24 Jun 2011 21:16:00 -0400

We are sponsoring Game Hack Day this weekend, and will be doing a short presentation about some secret projects we’ve been working on.

It’s not too late to register — http://gamehackday.eventbrite.com/

(score a free ticket and) Join OMGPOP at GoRuCo 2011!

Tue, 31 May 2011 11:06:00 -0400

The Gotham Ruby Conference (GoRuCo) is a one-day event that will take place in New York City on June 4, 2011. It is a technical conference aimed at highly motivated programmers interested in all things Ruby.

You’ll spend the day among Rubyists, Rails developers, and language enthusiasts in New York City, home of Silicon Alley. The agenda will feature 7 speakers and an open session of lightning talks by attendees.

10:00 a.m. - Few Constraints More Concurrency, Ryan Smith
10:55 a.m. - Less - The Path to Better Design, Sandi Metz
11:50 a.m. - Blow Up Your Views, Jeff Casimir
1:45 p.m. - HysteriaEngine: Ruby Does Violence in the Name of Science, Casen Rosenthal
2:40 p.m. - CoffeeScript for the well-rounded rubyist, Jeremy Ashkenaz
3:45 p.m. - Using your Database, John Crepezzi
4:40 p.m. - Build it and they will come (Rubinius Edition), Evan Phoenix

OMGPOP is giving away 2 tickets (valued at $199 each, no strings attached) to GoRuCo. This event is SOLD OUT, so this might be the only way left to attend this event. Why is OMGPOP giving two tickets away? As you may have read through the rest of our engineering blog, our entire API is written in Ruby. We love what Ruby does for us, and want to return the favor to the community.

If you are interested in scoring a free ticket, send an e-mail to chris@omgpop.com with the subject “GoRuCo Ticket” with a brief synopsis of yourself (or your website), and the winners will be announced no later than Thursday evening.
Good luck!

Update: We had a whole bunch of entries, but congratulations to our two winners —

Cedric Howe, Kickstarter
Serena Wales, Purpose

Thanks to everyone for expressing interest, next year we’ll have to give out more!

short blog post: use the mysql2 gem. (updated)

Fri, 20 May 2011 15:35:00 -0400

Step 1

# gem install mysql2

Step 2

a day later, we can see a more obvious impact:

Follow Up: New Relic, Redis and Resque (and now also redis-repeater)

Thu, 10 Mar 2011 16:47:40 -0500

It’s been exactly a month since we posted about solving production-woes with iowait blocked Redis. If you haven’t yet read that — please do: Praise for New Relic, Redis and Resque.

In that blog post I talked about what our next steps would be:

Going forward, we are planning to implement a non-blocking local buffer (another level of queueing) before being put into resque to reduce the latency of the API to the clients, which in turn gives an overall better user experience.

Well.. fate stepped in, and while at the NYC Ruby meetup on Tuesday night which had a focus of Resque, John Crepezzi from Patch (your source for local knowledge you can’t live without) did a lightning talk about a utility he put together called redis-repeater. I won’t give too many details on how they use it at Patch, but for us at OMGPOP it made perfect sense. In the possible scenario that our redis server becomes iowait bound, which we have unfortunately again seen a few times since our change to no longer persisting to disk, our main API will become dependent on waiting for this Resque enqueue method to return. To quell that iowait, and rather than implementing a “non-blocking local buffer”, we theorized that using redis-repeater we could put in another Redis ‘hop’ before the main Redis machine. This piqued my interest, and so I went back to the office to put theory into proof of concept.

I ran into a few issues with redis-repeater so I made some modifications which led me to doing my first ever (personal) GitHub pull request for, but not longer after that we had a working redis-repeated resque hop.

Unfortunately, Wednesday was spent working through JS/CSS bugs in IE7 for a new awesome game we are making, but we managed to work in the New Relic redis intrumentation from their rpm_contrib gem. The benefit, beyond seeing the duration and type of all Redis calls, is that all Redis calls are now registered as “Database” rather than “Ruby”. The large spike of “Ruby” in the chart from the previous blog post would have been “Database” which now would give us quicker insight into what the problem could be.

In the breakdown of a web transaction, we now see this:

Now today is Thursday, and we have pushed the changes to use redis-repeater in production:

Of note is to mention that the goal of this change is not to greatly decrease the latency between the OMGPOP API Servers and the Redis server (it’s already less than 1ms), but mainly to keep it at that low latency. In the event that the main Redis server now gets backed up with jobs our API will not be harmed.

Before:

After (and now in production):

So far, we’ve already `repeated` more than 3 million items (and Resque has now processed 9.78 billion items).

Next step (as made obvious by the above graphs): Distributed redis using Redis::Distributed

Thanks for reading!

Some links:

Come join OMGPOP at RubyNation on April 1-2, 2011 in Reston, VA

Thu, 03 Mar 2011 17:16:00 -0500

Ruby Nation is an annual two-day, dual-track technical conference presented by the Washington, DC area Ruby Community. The next RubyNation conference will be held April 1-2, 2011, at the Sheraton Hotel in Reston, VA.

We’re proud to announce that OMGPOP is a Platinum level sponsor of the event.

If you plan on attending don’t be afraid to get in touch with us at the event, we’ll make sure to bring some stickers! Tickets are selling out fast.. so get them while they are still available.

Links:

RubyNation
OMGPOP Jobs
Play our latest game on Facebook: Draw My Thing!

Migration of some core services to SoftLayer's CloudLayer.

Fri, 18 Feb 2011 18:28:00 -0500

In August 2010 we moved our physical infrastructure from Rackspace to ThePlanet, and this past November SoftLayer completed their merger with ThePlanet; for multiple reasons, this was great news to us. You can read more about the merger over on Tech Crunch.

As the article states, the new SoftLayer now provided us additional benefits:

An expanded product line, with new products and services not available to you before.
Industry-leading automation, Customer Portal, and Open API, for direct access to more than 150 backend systems and activities.
Increased geographic diversity and the ability to choose where your servers reside.
High-speed network and multiple PoPs, providing more than 1,500G of connectivity and direct connections for lower latency.
Exclusive network architecture that weaves together distinct Public, Private and Data Center-to-Data Center networks.
Improved Service Level Agreement with 100% uptime and 2-hour or less hardware replacement (failure or upgrade) guaranteed.
Greater value for your business, including more performance per dollar, a larger international presence and expanded partnerships with industry leaders.

One of the changes I noticed immediately was SoftLayer’s Customer Portal. It’s very intuitive and everything you need to do is within a few clicks — sometimes I get nervous I might impulse buy that server I always wanted, it’s too easy. The Order Server portion of the Portal reminds me of when I was younger and would price out the must ridiculous computers on Dell’s product configuration website of yesteryear, only this time when I click order it gets turned up in only a few hours (and it’s actually a quality product — sorry Dell!). We had previously built our own server management portal off of ThePlanet’s beta RESTful API, but for now SoftLayer’s Customer Portal is good enough that we don’t need to migrate that over to using their new API.

So what’s the next thing I did? Turned up a whole army of CloudLayer instances! CloudLayer is SoftLayer’s implementation of on-demand and scalable computing. So far we’re using CloudLayer for production level services such as an XMPP Cluster and Flume/Hadoop/HDFS Cluster and many development projects such as a node.js farm. Previously some of these services existed in Amazon EC2, but the ability to choose the specific amount of CPU Cores and RAM resulted in great cost savings compared to using the limited amount of instance types that Amazon provides. Additionally, latency between our physical infrastructure of servers at our Dallas co-location facility (via ThePlanet) to SoftLayer’s CloudLayer facility (our choice was naturally their Dallas facility) reduced to 6ms from 40ms.

We plan on migrating many of our other core services to CloudLayer as soon as possible, as well as much of our existing physical infrastructure over to SoftLayer’s new “Dallas 5” facility. These days, the ability to exist in such a hybrid environment is key to the scalability of most web applications, and we’re happy to start playing with the new possibilities that SoftLayer brought to our doorstep.

As always, if you’re interested in joining OMGPOP we are hiring — JOBS @ OMGPOP.

Praise for New Relic, Redis and Resque.

Thu, 10 Feb 2011 11:54:00 -0500

New Relic strikes again! Kudos to New Relic’s RPM for helping us isolate an issue in production.

We noticed an increase in latency (12 -> ~25ms) over the past week and did a deployment this morning (marked by the vertical line) to see if it would fix the issue.

Unfortunately it didn’t fix the issue. We did some additional troubleshooting and tracked the slowness down to only happening on calls that were using Redis for Resque jobs. Turns out, redis was recently reconfigured to persist to disk every 10 minutes (configuration: save 600 100000). This is completely the wrong configuration to have when using resque at the level we use it. Our resque environment handles over 50 million queue jobs a day. In the 10 minutes it will take to write this blog post it will have completed nearly half a million jobs.

That’s quite a large amount of throughput, and it makes no sense to be writing all these changes to disk. This led me to make a change in redis’ configuration which caused additional high load on the systems. This is reflected by the large spike after the deployment, which confirmed my belief that redis was experiencing increased latency of enqueuing items via resque. The fix? Good question. There are multiple ways to fix this scenario — turn off persisting to disk entirely, increase the save configuration flag to a higher interval, decrease the IO latency when writing to disk (faster disks), and many others.

For now, I changed persisting to disk to once an hour, and as you can see the change was immediately reflected. Going forward, we are planning to implement a non-blocking local buffer (another level of queueing) before being put into resque to reduce the latency of the API to the clients, which in turn gives an overall better user experience.

Onward we scale!

If you haven’t yet played around with New Relic I believe they have a free version which could be of great use to you — we at OMGPOP can’t recommend it enough.

Links:

facebook's Hacker Cup: the fourth question

Wed, 12 Jan 2011 18:13:00 -0500

The facebook Hacker Cup is an annual facebook programming competition where hackers compete against each other for fame, fortune, glory and a shot at the coveted Hacker Cup. A few of the software engineers here at OMGPOP decided to participate in this year’s Hacker Cup. The qualification round had three questions, each with varying difficulty and methodology. Each of us from OMGPOP that participated met the requirements to continue on past the qualification round, and after seeing the scoreboard [link] our eyes lit up as if we were presented with a fourth question.

Out of 98,348 people who RSVP’d for the Qualification Round [link] only 5,846 qualified to continue. That’s a lot less than any of us expected. What stood out most is only 527 correctly answered each of the questions. That’s only 0.54% of the initial group! We then wondered: What noteworthy analytical data can be inferred from the scoreboard of the qualification round? This would be our fourth question.

We started by extracting as much authoritative data as possible from the scoreboard. In the true fashion of facebook (or at least the film the social network), we simply curl’d each scoreboard page and parsed out the following tid bits of information:

rank
image url
name
penalty
p1 (problem 1: correct/incorrect)
p2
p3

p1/p2/p3 refers to questions 1/2/3. correct means they had a check in that column, and incorrect means they had a dash (note: incorrect may mean they did not submit an answer).

image url refers to the profile picture shown between the rank and name. The example for me (cwh) would be: “http://profile.ak.fbcdn.net/hprofile-ak-snc4/hs714.ash1/161268_8802104_6084533_q.jpg”

which is: (flower in my ear was from peer pressure)

What’s interesting is that you can determine my facebook user id from that image url, my facebook user id is 8802104 (see it in the url above?). This is where this question became fun. Rather than just stop there, we used facebook’s graph api to gain a few more pieces of data (this is public information) based on each user’s facebook id:

gender
locale (country)

We put all the data together, and came up with some interesting analytical charts.

Hacker Cup Qualifiers Avg Rank by Country [link]

Here is the averaging ranking of countries for the hacker cup. In red shows the average rank, and in white is the number of participants in that category. Some questions that come to mind… has china surpassed the US? Has communism beat democracy? Of note is that Estonia now has a stable internet connection, and even the Vatican brought its blessed coders to the challenge.

Hacker Cup Qualifiers Gender by Correct Answers [link]

Here’s a breakdown of how each gender did on each question. P1 refers to the first problem, P2 to the second problem, etc. We looked into the lone female who got just problem 2 correct, and posted her picture on here for novelty and viewing pleasure (however, there are rumors that maybe a friend submitted through this account).

Hacker Cup Qualifiers Gender Breakdown [link]

Most people who qualified were guys. Nuff said. The first female appears around rank 70ish according to the data.

Hacker Cup Qualifiers average facebook id [link]

No matter what anyone tells you, how big your facebook id is matters. This graph proves it, definitively (and arbitrarily).

facebook id vs. hackercup qualifying rank [link]

The above chart and next are just looking at the distribution of the users facebook ids and how well they did. While this may not be interesting from a hacker cup perspective, you can notice the apparent ‘gaps’ between users facebook ids. Probably, facebook decided at some point (maybe coinciding with the move to use big-ints as ids instead of integers) that ‘all new users ids should start at 1,000,000,000’. Then, looking at the next picture (and after significant Facebook growth) they did this again at around 100,000,000,000,000.

for people with large facebook ids [link]

Hacker Cup Qualifiers Avg Rank by Gender [link]

In this graph, we have an arbitrary trend line on a bar graph that hints somehow that if your gender cannot be gleaned from the graph api OR you didn’t have a public photo, that your average rank is lower than that if you are a male or a female. Another way of saying this is that if your gender is somehow ambiguous, that you have super facebook hacking powers — data doesn’t lie.

If you have any other ideas for analytical charts, here is the raw data (in excel format). We’d be happy to post any of your findings, and don’t forget.. OMGPOP is hiring!

kudos to cwh, nartzpod and andweez!

OMGPOP is the best place to play fun multiplayer games online, for free. We have a diverse portfolio of games, both on our own platform and on Facebook, and develop them all in-house.