cloudmagic analytics uses mongodb

MySQL, the world’s most used open source RDBMS has been one great success story. And now, a new breed of database management system – NoSQL, which has no adherence to the widely accepted RDBMS, is showing some promise. One such project is MongoDB, part of the NoSQL family of database systems.

With developers looking for alternatives to the traditional relational database, MongoDB (one of the fastest growing alternatives) offers a number of compelling advantages over the others. Replicating the MySQL success story can be really hard for MongoDB, and with popularity comes a lot of criticism. Robert Scoble summed it up nicely in his comment: the best technologies gather passionate haters!

Does CloudMagic use MongoDB?

We do not use NoSQL at the heart of the product, but we did happen to find a valuable use case of MongoDB – running our backend analytics. It gives a good combination of flexibility and ability to query the database (though the querying capability can not be compared to a RDBMS, it’s still quite useful).

How we do it?

We parse the logs using shell commands and PHP to form a JSON and then dump it into a MongoDB Collection. You might be wondering why MongoDB as this could have been achieved using MySQL (or any RDBMS) itself. Well, the major benefit is that no two requests are alike, so the JSON formed does not have the same keys. This is where the flexibility of NoSQL comes in handy. With no concept of column names here, all you have to do is to insert a JSON in a collection and you are done.

Why MongoDB?

One can also create a key/value table in MySQL, but imagine having 200,000 requests per day with approximately 30 key/value parameters, viz 6 million rows. It’s just difficult to get desired results from such a table using a single query, you will need to use multiple joins of the same table. Whereas in MongoDB you can get the result in a single command. Speaks volumes of what MongoDB is capable of.

Map/reduce gives another edge, MongoDB uses JavaScript as query language thereby allowing us to write functions and manipulate the data in the database server itself. Map/reduce is actually quite powerful, you can implement many custom conditions to handle the data and it can also act as an alternative to GROUP BY in SQL.

To get the analytics for a given day, we query all the requests for that day and get certain numbers like ‘Daily Active Users’, ‘Number Of Searches’, ‘Number Of Previews’ etc. We do create intermediate MongoDB collections for simplification so that each request follows a certain standard. This intermediate collection holds information of selective request types like search, preview, snapshots which makes it even more manageable.

With each CloudMagic app upgrade or feature implementation, the request data and format might change, but there is no major change in the analytics database, this makes a our lives a lot simpler.

Fascinated by MongoDB? We sure are. Share your experiments with MongoDB. Drop a comment below or join the discussion on hacker news.

Image courtesy: Junya Ogura on flickr

  •     •     •

If you are looking for a nice email app that works across iOS, Android and Mac, give Newton a spin!


Written by

Rishabh Dua

Rishabh manages AWS Infra & Analytics at CloudMagic.


Leave a Reply

Your email address will not be published. Required fields are marked *