Monthly Archives: November 2012

Simplicity in Software Architecture is Underrated

fight hard against complexity

“Simple is better than complex.
Complex is better than complicated.” – Zen of Python

So what makes something Simple, Complex or Complicated?

It varies – A problem could be simple if you are Einstein but unfathomable for many of us. In case you are on the other side: hi Einstein!

Generally speaking, a problem is simple if it has just a few components and our mind grasps it easily. A complex problem can have many components and our mind takes some time to understand it, say, 20 minutes or more just to understand the problem. A complicated problem is a beast (sometimes beasts). You will keep missing few parts of the problem even after you get it. It is very difficult for our mind to keep the entire context of such a problem.

Why bother about it?

We are software developers and we love solving problems once and for all so that other humans won’t have to. The more complex a software becomes, the more difficult it becomes to maintain. This makes the software more likely to break due to changes and difficult to debug. It can cripple the team and bring the development pace to crawl.

Complexity is inevitable

Every non-trivial and useful software will gain weight as it keeps expanding its problem horizon. That is to say, complexity is inevitable for such a software because we will keep adding features and enhancements to make it more useful to our dear users.

Why should we fight against architectural complexity?

Since architectural complexity of a software already depends on the complexity of the problem it solves, what we should not do is make the software unnecessarily complex. As obvious as that sounds, it is important to keep it in mind, because it’s very tempting to step over the line and make something more complex than it should be. Why? Kendra Cherry has explained it better in this article. Though all the four “obstacles in problem-solving” listed in that article are very relevant for software developers, I must mention two of them right here. These are faced very frequently in our field:

Assumptions: When dealing with a problem, people often make assumptions about the constraints and obstacles that prevent certain solutions.”

You bet. We assume 100,000 users are going to sign up within a couple of days after we launch. We assume that a simple network call will take so long that we have to keep the data prefetched and maintain a cache. We assume a lot of such constraints and add unnecessary complexity to the architecture which is not needed right at this moment. According to Kendra the second obstacle is:

Irrelevant or Misleading Information: When you are trying to solve a problem, it is important to distinguish between information that is relevant to the issue and irrelevant data that can lead to faulty solutions. When a problem is very complex, the easier it becomes to focus on misleading or irrelevant information.”

I couldn’t agree more. It is easier to focus on these questions instead of the real problem at hand: should I be using NoSQL or MySQL? If NoSQL, will it be MongoDB or CouchDB. Enlighten me but I have never heard of a software that failed because it was using CouchDB instead of MongoDB (or vice versa).

Keeping things simple is hard, but important. While architecting and developing a software, you have to be careful about everything you are adding to it. Always ask yourself – Is that absolutely necessary? I would like to end this post with the following quotation:

“It is not a daily increase, but a daily decrease. Hack away at the inessentials.” – Bruce Lee

P.S. Please go and read the whole Zen of Python if you have not already.

Image Credit: dno1967b on flickr.

CloudMagic Analytics – MongoDB finds its place

cloudmagic analytics uses mongodb

MySQL, the world’s most used open source RDBMS has been one great success story. And now, a new breed of database management system – NoSQL, which has no adherence to the widely accepted RDBMS, is showing some promise. One such project is MongoDB, part of the NoSQL family of database systems.

With developers looking for alternatives to the traditional relational database, MongoDB (one of the fastest growing alternatives) offers a number of compelling advantages over the others. Replicating the MySQL success story can be really hard for MongoDB, and with popularity comes a lot of criticism. Robert Scoble summed it up nicely in his comment: the best technologies gather passionate haters!

Does CloudMagic use MongoDB?

We do not use NoSQL at the heart of the product, but we did happen to find a valuable use case of MongoDB – running our backend analytics. It gives a good combination of flexibility and ability to query the database (though the querying capability can not be compared to a RDBMS, it’s still quite useful).

How we do it?

We parse the logs using shell commands and PHP to form a JSON and then dump it into a MongoDB Collection. You might be wondering why MongoDB as this could have been achieved using MySQL (or any RDBMS) itself. Well, the major benefit is that no two requests are alike, so the JSON formed does not have the same keys. This is where the flexibility of NoSQL comes in handy. With no concept of column names here, all you have to do is to insert a JSON in a collection and you are done.

Why MongoDB?

One can also create a key/value table in MySQL, but imagine having 200,000 requests per day with approximately 30 key/value parameters, viz 6 million rows. It’s just difficult to get desired results from such a table using a single query, you will need to use multiple joins of the same table. Whereas in MongoDB you can get the result in a single command. Speaks volumes of what MongoDB is capable of.

Map/reduce gives another edge, MongoDB uses JavaScript as query language thereby allowing us to write functions and manipulate the data in the database server itself. Map/reduce is actually quite powerful, you can implement many custom conditions to handle the data and it can also act as an alternative to GROUP BY in SQL.

To get the analytics for a given day, we query all the requests for that day and get certain numbers like ‘Daily Active Users’, ‘Number Of Searches’, ‘Number Of Previews’ etc. We do create intermediate MongoDB collections for simplification so that each request follows a certain standard. This intermediate collection holds information of selective request types like search, preview, snapshots which makes it even more manageable.

With each CloudMagic app upgrade or feature implementation, the request data and format might change, but there is no major change in the analytics database, this makes a our lives a lot simpler.

Fascinated by MongoDB? We sure are. Share your experiments with MongoDB. Drop a comment below or join the discussion on hacker news.

Image courtesy: Junya Ogura on flickr