A Year with MongoDB

This week marks the one year anniversary of Kiip running MongoDB in production. As of this week, we’ve also moved over 95% of our data off of MongoDB onto systems such as Riak and PostgreSQL, depending which solution made sense for the way we use our data. This post highlights our experience with MongoDB over the past year. A future post will elaborate on the migration process: how we evaluated the proper solutions to migrate to and how we migrated the data from MongoDB.

First, some numbers about our data to give context to the scale being discussed. The figures below represent the peak usage when we were completely on MongoDB — the numbers are actually much higher now but are spread across different data stores.

  • Data size: 240 GB
  • Total documents: 85,000,000
  • Operations per second: 520 (Create, reads, updates, etc.)

Read More

EC2 to VPC: Executing a Zero-Downtime Migration

A couple weeks ago, Kiip completed a transition from EC2 to VPC. In a previous post, I talked about the benefits and basic terminology of VPC. In this post, I’ll cover the planning that went into it, some software we’re using with our VPC, and executing our planned migration with zero downtime.

Transitioning from EC2 to VPC is not like transitioning from one set of EC2 machines to another. It’s more like transitioning from one hosting provider to a completely new hosting provider. The main reason for this is because nodes outside the VPC cannot talk to nodes inside the VPC (with exceptions, of course). Therefore, you need to almost bring up an entirely new cluster alongside your existing cluster, and make a big switch to the new machines while making sure everything continues working smoothly. All the while, you need to make sure you have no data loss during this process.

Read More

EC2 to VPC: A transition worth doing.

Kiip recently completed a migration from EC2 to VPC. VPC exited beta and became generally available in all regions in August, and allows you to provision compute nodes within a virtual network in AWS. For anything more than simple websites, we believe migrating to VPC is something worth doing, or at the very least worth investigating.

Because Amazon VPC is a new service and requires a substantial amount of domain knowledge, this article will first cover a quick intro to the benefits and parts of building a VPC. Specific details about our VPC architecture here at Kiip, tooling we’ve built around it, and our migration process will be covered in future posts.

Read More