We have spent much of the last four years building our factory in the cloud and ran into many challenges along the way. As a victim of our own success, one of those challenges last year was when the volume of work required us to scale and optimize our technology platform faster than we ever anticipated.
Although we adopted a “service oriented architecture” (SOA) when making the move to CloudFactory 2.0, we never knew the challenges in store for us. We did not fully segregate the system components as dictated by SOA methodology which we only realised later on. When we kicked off 2.0 we had no real data in the platform and everything was a breeze early on.
As we grew our workforce from 300 to 3000 cloud workers, the lookup/fetch time to assign a microtask also grew and the system started to respond slower than we and our cloud workers would like. We tweaked, experimented, hacked, and optimised to keep up with the growing demand but the response time was still not up to par. It demanded multiple engineering brainstorming sessions to rethink many things and it boiled down to our task dispatcher which has proven itself as one of our key components within the platform.
We decided to decouple this task dispatcher from our core platform and use Redis (RAM database) as our backend technology. Decoupling this way helped us to make it more cohesive and focus on its sole job i.e. task dispatching. This also made it very light-weight and blazingly fast. Actually, its now a smaller sinatra web app with a handful of private APIs. Using Redis helped to boost the performance of the app and we saw the results right away. Yay Redis!!! It is simple to setup and super FAST. It is a very handy tool to store transient data that need fast access. And it exactly fits in our context.
The day we went live with the new task dispatcher we saw a dramatic reduction in the response time to assign microtasks to workers. But here is a visual to prove it!
Like many of our clients, we are also faced with challenges and bottlenecks in our own growth. And it feels great to continue working together with the latest technology and innovative thinking to break down these barriers and take everything to the next level. Our team of 43 engineers are doing this on a weekly basis and we celebrate their wins (big and small) because they add up to happier cloud workers and happier clients!
(CloudFactory's Sr. DevOps Manager, Kaji Bikash and Software Architect, Poudel Bikash contributed to this blog.)
