At d.labs we aim to share knowledge across our engineering teams. We are aware that constant technical education is a must in our industry for each individual regardless of his role. So there can be a back-end or front-end developer, tester or QA engineer, Dev-ops or designer and Product managers. Everyone, based on their daily tasks can grow by following the latest industry standards and latest approaches, technologies, processes or tools that can make their work constantly improved.
On the other hand we have a lot of projects and different types of products that are solving different business problems, targeting independent segments, are written in most-suitable programming languages, or run on different technologies. Each of them has, of course, because of that, strikingly different challenges, technical or process vise. Sharing that experiences and knowledge across teams or squads, as we call our engineering teams, is crucial. This is also one of our opportunities to see how on one side, our clients can benefit faster and better solutions and on the other side, engineers not being stuck with constant daily tasks but growing in a collaborative working environment.
Just recently one of our squads pinpointed an issue with their product - a bottle-neck on their daily performing cron jobs that were run for thousands of users each day. The team which is developing that product, assumed where the issue was, but together with tech lead engineer and Squad lead they came to me, introduced the issue and the reason why it would be better if some experienced engineer from other team would tackle it and how his unbiased approach would probably provide better and faster results instead of developers that are actually developing that part of the product.
After initial conversation and aligning on resources, one of our senior engineers was involved in this cross-squad collaboration and this is how he described this project from his perspective:
“One of the cross-project's requirements was to encrypt certain fields in the database. What seemed the best and fastest way to implement at the time was just to add the php library which adds data encryption at ORM entity level. It is not the ideal layer to use encryption in, but it was the way the chosen library did it.
This worked OK on a small set of fields but the time increased significantly when there was a bigger set of fields that needed to be encrypted or decrypted. CLI commands that should run for a few minutes were now running for hours. Something needed to be done and this was the point where I was asked for help.
After doing an initial research to pinpoint the actual problem I found out that the library that was used actually uses another underlying library which does the work of data encryption/decryption, but the one that was used has implemented encryption algorithms in pure PHP. This means that PHP does all the heavy work of complex calculations and it is really slow at doing heavy math.
After confirming that encryption is the point where the majority of execution time is, I did research on what the options to replace the underlying library for data encryption are. Solution was to replace it with a new library that offloaded encryption to the PHP extension, which is basically just a wrapper around another C library for encryption and decryption. After replacing the library, the execution time dropped significantly, commands were executing up to 40 times faster, so something that was running for hours now ran only a few minutes.
To finish the complete solution, the whole database needed to be decrypted with the old library and encrypted back with the use of a new faster library. Decrypting the whole database took around 10 hours while encrypting it took only around 15 minutes.
At first I was kinda skeptical and thinking why do I need to go and help on some simple thing like find why something is slow. One thing that senior developers don't like is to solve small problems especially when they might look like something caused by lack of experience. It might feel like you're cleaning the room after a kid. But what we forget too often is that we were also once at a point where we lacked experience and maybe not so lucky to have somebody help us and we were stuck for hours trying to solve something. But why shouldn't we now spare others and share our experience with them so they can build new and better things on existing knowledge. There is this good feeling where you know you helped somebody become better at something or at least you made their experience somewhat better.”
I was really impressed to hear this. This great example of collaboration across teams emphasized not just good technical solutions in the end but especially a sense of achievement on all levels.
Engineer who supported the external team by gaining new experience, the team that worked on a product by receiving not just quick and good solutions but also external opinion and suggestions on approaches that they are using, our client to have a better and more reliable product and of course d.labs with another great team collaboration and engineering cognitions which we all salute.
I am proud of seeing those kinds of real life examples in our tribe and I am looking forward to experiencing them more and sharing them with the rest of the world.