They say experience is your best teacher. And we at Lemberg Solutions totally agree.
But who said you should necessarily learn the hard way? Our Cloud, Embedded, Data & AI, and Digital department leaders share real-life f*ck ups they encountered while delivering projects to our clients.
Sounds interesting? Dive in and discover solutions to the toughest mid-project failures.
Cloud engineering mess-ups and their solutions
Let’s review more examples of failures from our Cloud team’s experience and discover the solutions they utilized to tackle the challenges:
Ignoring scaling costs
What happened? The client launched a demo of an IoT platform consisting of 500 devices, expecting a low data load. The client’s team planned for each device to send data once a minute, which was perfectly fine with their free AWS plan. However, after the launch to the market, the data exchange rate increased to once per second. This led to unexpected AWS charges and resulted in slow dashboard load time.
Solution: After conducting a thorough performance audit, we delivered an optimized data management plan. We introduced data caching mechanisms to reduce the data load. We also provided the client with information on the importance of scaling costs and projections for different system use levels.
Incomplete project information
What happened? One of our healthcare clients had a negative experience building their product before they came to Lemberg Solutions. Lacking information about their project initially resulted in building a product demo without key features and functionalities. Incomplete documentation without clear user flows and requirements confused their former vendor's team. Unfortunately, our client didn’t receive all the necessary guidance from their first vendor, so they couldn’t collect all the necessary data.
Solution: Lemberg Solutions experts informed the client of the initial solution discovery and its importance for the product's success. We guided the client through the discovery process and, eventually, received all the necessary information for building their product. The process included discussing business goals, detecting user personas, outlining and structuring solution architecture. For more details on the discovery phase process, check out our solution discovery page.
Underestimating multi-component database complexities
What happened? Our client’s system consisted of 10 separate components. However, they didn’t pay enough attention to the strong dependencies between these components. It led to blocked releases and scaling challenges. While introducing changes to one component, the other were also affected, causing prolonged release cycles. The same goes about scaling — the entire system had to scale as a whole, but our client ignored it, only scaling some of the components. As a result, the system was using costs and resources inefficiently.
Solution: Our team redesigned the system architecture to decouple dependencies. This enabled independent releases of system components and minimized downtime. We also introduced containerization and auto-scaling policies to make sure the client can scale components based on real-time demand.
Test your intuition — which mistake led to one of the largest data leaks in history?
Embedded engineering failures and our ways out
The flow of an embedded product development process is often hard to predict, as several external factors, such as hardware supply shortages or defective components, often complicate it. Keep reading to find the solutions our embedded engineers came up with to deliver projects regardless of challenging factors:
Library and manufacturing flaws
What happened? Libraries from hardware manufacturers often don’t perform as expected, which may lead to delays and over-budgeting during the engineering process. On one of our projects, we faced an issue — the software library provided by the hardware vendor wasn't working as expected and did not cover the full functionality of the hardware component. The chip didn’t operate as expected; moreover, communication with the chip collapsed regularly.
Solution: Our engineers responded to the detected issue immediately. As the project was time-sensitive, we worked together with the hardware manufacturer to fix detected issues and add missing functionality. As a result, we fixed the malfunctions and delivered the project on time.
Incomplete or changing requirements
What happened? The client requested a fixed-cost estimate. We provided it, and after the client’s approval, we launched the project. Then, the client informed us that some critical functionality was missing. Though we managed to introduce this feature within the initial budget and timeframes, the client kept asking to add more features within the existing scope, slowing down the development process and postponing the project delivery.
Solution: Continuing with a fixed-cost approach was pointless. Thus, we switched to an hourly-based model and started planning sprints together. This way, our client gained the flexibility necessary for their project.
Hardware shortages or malfunctions
What happened? We were building an embedded product that relied on a specific type of microcontroller. Halfway through the project, our component provider announced a shortage of their microcontrollers that would disrupt their supply chain. The delivery of microcontrollers we chose would, at best, take six months, significantly delaying our original project timeline.
Solution: As we were already in mid-project, our team had to take action immediately. To solve the issue of hardware shortage, our engineers called an urgent meeting to identify alternative microcontrollers with similar capabilities. Next, we adjusted the firmware to the new components, releasing the project only two weeks behind the original schedule.
Ready for the next question? In 2016, a major "IoT botnet attack" took down the internet in several regions across the globe. What was the underlying cause of the failure?
Data & AI engineering screw-ups — how did we turn things around?
Check out the software engineering failures and their solutions from our Data and AI team below:
Undefined roles and lack of technical leadership
What happened? Let’s review a specific case where the project roles were poorly defined on the client’s side. Our client was building a product that combines AI and software development. However, they didn’t have a technical lead who could manage these two directions, taking into account the specifics of each. This led to miscommunication, poor implementation, and a general sense of chaos.
Solution: We helped the client structure and define roles and responsibilities from the start of our cooperation. Having 200+ experts on board, providing a candidate with experience managing cross-functional projects took us less than a month. From that moment, the project moved on smoothly.
Insecure data sharing
What happened? While working on one of our Data and AI projects, we were surprised to find that the client’s employees shared credentials and other critical data types through insecure channels, like Slack and emails. Having no idea it increased the risk of data leaks and unauthorized access, our client’s employees exchanged sensitive data using personal communication channels.
Solution: The Lemberg Solutions team helped our client establish secure data-handling protocols. We introduced encrypted communication channels and proper access controls, ensuring higher data security. From that point, the data security bar has been set high for our client.
Poor data quality
What happened? Our client had a vast dataset they were looking to use for the development of an ML model that would optimize their business processes. However, we soon learned their data was filled with duplications, missed values, mistakes, etc. It was preventing the possibility of building an algorithm that would work for the benefit of the client’s operations.
Solution: Setting up well-structured data collection from the start is the key to developing accurate AI systems. In this case, we ran the data cleaning and labeling process. After deep data cleaning, we could only use 3% of the dataset for training the client's ML algorithm. Then, we configured data validation rules to ensure the client’s data was further collected correctly.
Time for another short quiz! Which of these AI-related failures was caused by biased data used for training an algorithm, leading to false outcomes?
Digital engineering failures and ways to pull through
Below, our Digital team reveals this and other fails they faced while working with our clients. Keep reading to find the solutions our team members came up with to resolve the issues:
Inefficient migration process
What happened? We found hidden features and unaccounted content while migrating our client’s website from a legacy platform. This set off a chain of delays, as instead of following the regular migration plan, we had to solve the newly defined issue with unarranged features and content.
Solution: After defying the purpose of the hidden features and integrating unlogged content into the system, our team established a plan for regular audits, regression testing, and stress tests to avoid similar surprises during future migrations.
Neglecting design refinement
What happened? Our client built a mobile app using the services of a third-party vendor. Eager to launch the app as soon as possible, the client insisted on using the basic wireframes their team had designed previously. After entering the market, the app received negative feedback from users. Poor navigation and shifting fields were just a few complaints from the users.
Solution: With a long-standing experience in app development, the Lemberg Solutions team advised the client never to avoid the prototyping stage. As designs are constructed statically, they may fail the live app without proper prior refinement and testing. We set up the design and development processes from scratch and rebuilt the app using well-tested designs, corresponding the exact app users’ needs.
Unclear information architecture
What happened? Our client assumed their users would intuitively find the needed information even in the most hidden corners of their website. Thus, they never considered adjusting navigation or updating the website according to user needs. The results were disappointing — website visits and session duration kept dropping constantly. That’s when they decided to seek help from an experienced vendor.
Solution: We helped the client by introducing a user-centric design approach. Our team also integrated usability tests into the client’s development process. For example, our engineers ran a blind navigation test on the client’s website. Blind navigation tests are usability tests in which participants navigate a website or an app without prior knowledge of its structure or features to provide an objective evaluation of UX.
Lack of QA investment
What happened? The main mistake of our client, a middle-sized ecommerce platform, was thinking that "developers will test everything, we'll figure it out as we go." They built the first version of their website using this approach, and even before the release, the client realized something was off. The payment gateway and filters were malfunctioning, the website worked on Chrome but had problems on other browsers, the poor mobile experience with overlapping elements, and dozens of other issues made the website launch impossible.
Solution: The client came to Lemberg Solutions with a request to fix their website. At that point, they had already realized that cutting the budget for the QA needs was their critical failure. Right from the start of the project, we established a well-structured QA process and introduced a plan for regular regression tests.
Let's hit the last question! Which of these failures was caused by a bug in the software during a space mission?
Key takeaways
After going through all these cases, we hope you don't think we're attempting to scare you or suggest giving up on building your product. It’s quite the opposite! Though creating your solution might be rough, it’s always worth it.
To minimize the number of failures along the way, find a reliable tech vendor who’ll guide you through a safe path and help you build a product that meets your expectations.
