About the client
WeighMyRack “is for researching and comparing gear specs” as they position themselves. In other words, it’s a price comparison website. We met Alison Dennis - one of the founders - on Upwork at the end of 2018 and started working at the very beginning of 2019. She was looking for someone with profound Drupal expertise for a complex Drupal 7 project. The bottleneck was that the developers who had been supporting a website until that moment decided to cancel their agency so the replacement was needed as soon as possible. Having conducted a technical interview and a couple of chats, ADCI Solutions Drupal developers were chosen as the main website developer. Since then, we’ve been providing ongoing maintenance and enhancing the website.
Wait, what? Outcomes in the very beginning?
We understand that you may be short of time. So let us first talk business.
What did we improve?
First of all, we checked the code and found 9 functionally changed modules. We cleaned them so that we could do the upgrade and don't break the whole website. The thing is, the last security updates were 2-3 years old.
Secondly, we refactored the import system and decreased the update time by 2 times - from 26-28 hours to 12-15 hours.
Since we’ve added more retailers feeds, CX (customer experience) was improved: now users can compare prices in more stores across the world: Americas, Europe, UK, Australia, Canada.
That’s about figures and statistics. Now when we’ve got your attention, you’re welcome to read about the project’s specifics. Are you with us?
About the project
Now to the project itself. What is it about?
Any person that has a thing for rock climbing would like it without a doubt: it saves money by providing price comparison for different items. That is the business value that a website provides, and according to our data, WeighMyRack is successful.
The website has a rather simple and straightforward UX. In general, the main functionality covers interaction with products like the one in the picture below. All gear is divided into categories.
Within these categories, you can:
- find a piece of gear using the filters like a brand, price, color, shape, size, weight, awards won, discounts
- compare this gear by price in different stores
- give a product review
- add this gear to your list of gear
- add it to a wish list
- proceed to any online store from the website comparison list and buy equipment
Also, video reviews are available for many items.
Basically, the job includes scaling of import functionality (for 10+ external data feeds), Drupal updates, bug-fixing of existing functionality, technical support.
The client struggled to improve website performance, in particular - optimize the work and update of data feeds.
So, first of all, we started working on data feeds and proceeded to UI changes then: percentage off, filter on sales, sale badges, etc.
The main challenge was that there are two servers - the main one serving the users and the additional sync-server processing retailers feeds.
Another challenge was that there were some changed contributed modules at the very beginning of our work.
How did these 2 things put us in a bind? When it comes to servers, there can happen changes on each of them: for example, a content manager can apply changes on the main server while there’s an update on the sync-server. We have to save all the newly generated data somehow.
The hacked modules can simply crash all the website once we try to update them to the newest version. For that reason, we had to investigate how extensive the changes in the modules were.
Let’s talk about boring technical details a bit - they let us boost some business metrics after all. But if you don’t have much time - please, proceed to the Outcome section.
The servers system
As we already mentioned, there are 2 servers. The first one serves website users, the second sync-server deals with retailers’ feeds. These two servers are connected through Jenkins API - the system that optimizes development and deployment processes.
Why 2 servers? Well, everybody likes fast websites. If we had imports of retailers’ feeds happening on the main server, a visitor could spend the whole minutes waiting for one page to load.
The client and our team were worried whether a live website will have updated information from retailers feeds if any server goes down. It’s hard to make 2 servers work with a database in lockstep because this database is being constantly changed by one or another. We were afraid that one of the servers would fail and we won’t have relevant and updated information.
What did we do? We created a custom backup-storage on a remote server although we already had the backup_migrate module set up and weekly backups provided by the Digital Ocean hosting. That allows us to make sure that the updated information will be processed to production (the live website).
We also set up Monit - a system for monitoring and reanimating the processes. Before that, when the website was DDoSed, a database MySQL resisted to work because it couldn’t process queries. As a consequence, the website was down.
Now, if MySQL, Apache, httpd don’t respond within 3 minutes, Monit restarts the respective process/service and the website goes back to normal. It is especially helpful when a DDoS attack starts in the middle of the night.
The retailers feeds
Initially, there were just 2 retailers. So what the update process was like?
These 2 retailers launched at a particular time, one after another, and as soon as both imports are done, we merge sync-server and main-server database changes: we mean updated information about products from the sync-server and possible changes from the main-server.
How long these 2 feeds were updating? 3-5 hours on average. But the website was scaling, and we added more retailers - the update time raised accordingly. At one point, this total import time crossed 24 hours and the whole idea of every day update just failed, you know.
How we sorted it out? We separated the feeds from each other and started running them in parallel and simultaneously. Of course, we immediately faced some issues because, in Drupal, several queries tried to get access to the same information. Imagine yourself in a library: you want to take a particular book but it was borrowed so you have to wait. But Drupal doesn't want to wait and simply overrides changes made by concurrent imports.
In our case, when we tested updates for 1000 products, only 500-600 were updated. Do you think we gave up? No.
We implemented the lock mechanism: it allows keeping control over the data and doesn’t let other processes intervene until this data isn’t needed for a current process anymore.
Result: we decreased the update time for all the feeds from 26-28 hours down to 12-15 hours. Yeah, success!
Instead of conclusion
We hope you enjoyed this case study. There’s no feeling compared to getting success despite many obstacles.
Thank you for reading and we’re looking forward to telling you more stories! If your friend, business partner, relative needs web development help or wants to improve web app performance - just tell them that we offer a free consultation and are ready to help.