In case you’ve been looking for the best ways to eliminate bugs from your product before rolling it out, here’s some experience-based advice that will help you do it.
Why hardware and firmware testing matters
Delivering a well-tested hardware device on each release stage is one of the key success factors before starting the next product iteration. Initially, you try to impress your investors and/or backers with the first functional prototype, be it a PoC (validating technical feasibility) or an MVP (resembling the mass-production look & comprising core features) prototype.
Next, you give it to your beta users, and, of course, you want to get their feedback and build an upgraded version of your device. However, you need to be careful because you are dealing with hardware, and each new iteration can cost you more than you would usually spend with software-only products. This is why doing proper market research, validating your ideas, selecting the right hardware components, designing the whole hardware and software architecture, and writing test cases is very important during the early stages. To ease this process, we offer Discovery Workshops to scope out your work and plan the development and testing phases.
As mentioned above, you should start thinking about device testing as early as possible and include it into your timelines on the roadmap. Your product owner might already be aware of the rough numbers here. For example, the Lemberg team provides development estimates where manual QA testing hours are included by default, which equals 30% of development hours. More often than not, our clients ask us to add automation testing on top of the estimation request. Depending on the overall coverage, automation testing can add anywhere between 50-100% in development hours.
The first must-have you need to think about is software and hardware functionality testing. The core objective of hardware testing is to check the technology against its functional requirements. This is typically thoroughly documented in the functional specifications, acceptance criteria, and test plans that our QA engineers prepare during the workshop or just before we start actual development.
Every company aims to increase customer demand, which largely depends on the quality of their product. By testing your technological solution with the help of QA engineers, you get the opportunity to make sure that the key features of your device or application work as planned. The engineers also do Usability Testing to analyze whether users will be able to easily interact with your technology.
One of the main parts that can go wrong with your software includes user interface mistakes that don’t allow the user to operate the information they obtain. Another common problem is incompatibility between separate components of your software, meaning they work well independently, but cannot operate properly when combined.
To ensure effective IoT systems testing and debugging, you need to have a variety of programmers and a professional IoT lab with a relevant hardware infrastructure set-up, i.e. oscilloscope, signal analyzers, popular dev kits, and SBCs like Raspberry Pi or Arduino, etc.
Real-life cases
To reduce risks and unexpected expenditures for the clients, our QA team analyzes project requirements and use cases at the beginning of the development stage. We plan test cases and get all stakeholders on the same page from the get-go since some projects require specific equipment or devices that need to be acquired in advance.
Hardware testing
Hardware testing is all about finding out how durable your product is. Can it endure high or low temperatures and humidity? Will it work properly after some physical damage? Would it be comfortable for users? Our engineers test the device to see how well it will perform when being actively used. And these are only some of the things QA engineers check.
Livestock weight monitoring
Barkom Ltd. is a Ukrainian agricultural company that specializes in growing pigs and cattle and making food products for sale. They asked us to create a device that could weigh pigs without scales. Our team developed a technology that identified the pigs’ weight via a camera. We ensured system functionality by adjusting the device to the temperature in a pigsty and established remote access to check whether it worked properly. We recreated pigsty conditions in our laboratory and made a waterproof box so that the device wouldn’t break down. To avoid overheating, we integrated cooling fans into the box.
Lesson learned: When working with animals, you should consider the environment where data is collected. For instance, pigs prefer to assemble near the feeder. They also move constantly, which complicates data collection. Thus, we implemented the best practices of hardware testing by constructing a separate corral to measure the parameters of each pig.
Cell Garden: Smart hydroponic system for growing microgreens
Cell Garden is an IoT startup that developed a nutritional concept to make our lives healthier and better. When planning and designing this smart greenhouse system, our team had to account for multiple aspects to ensure effective testing. This includes a combination of lighting, water misting, drainage, ventilation, and, of course, embedded and cloud software that controls this whole system. Testing was split into 2 phases. We started off by running functional tests using dev kits and later first PCBs. When all the mechanical parts were printed we moved on to the next phase and were able to test the device holistically, including touch UI panel and all moving parts like drawers, water tank, buttons, LEDs, etc. to conduct usability testing and do a final run of hardware functional testing.
On top of that, our team had to ensure an even distribution of water through the nozzles. One important condition we had to fulfill was to maintain a very specific distance between the nozzle and the ground so that the water could reach every plant without damaging them. So we decided to purchase RFID modules to establish the required distance for the nozzles. This decision helped to implement a technology that would identify whether the position of the nozzle was correct.
Lesson learned: To find bugs early on, you can plan and conduct early functional testing during bring-up phases using dev kits and first PCBs even when the manufacturing stage of your device hasn’t been finalized. Functional testing of embedded systems will often help you to come up with an unconventional idea of how to make your device work.
Cavamo: AI-driven horse stress monitoring solution for SportsTech
Cavamo is a UK tech startup aiming to create a device that helps to maintain effective interaction between a horse and an equestrian. We ran sport tech device stress and stability testing in the Lemberg IoT Lab to make sure that the appliance would not overheat. Along with that, we checked the methods of its fixation to the bridle to ensure that it would be firmly attached to a horse in real-life conditions.
Lesson learned: You should check whether the device will endure harsh weather conditions if it has to be used outdoors. Don’t limit your investigation to humidity testing only.
Hygiene monitoring and activity tracking system for Agritech
Our client, Barkom, asked us to create a device that would help them track whether their workers maintain a high hygiene level. This is important for keeping the pigs healthy. We created an artificial shower room in our laboratory to check how the device will perform in such conditions. To find out whether the device would work on the farm, we decided to test it on-site. Our team noticed that the structure of the walls in the pigsty differs from those in the office. The difference influenced the accuracy of signal transmission, so we had to cover the farm walls with metal sheets.
Lesson learned: You should consider the density and composition of the environment in which your device will transfer data to the cloud. In some cases, the chemical composition of certain materials impedes the technology from transmitting the signal..
DAV: IoT module for micro-mobility platforms
DAV Foundation is a blockchain-based transportation company that asked us to develop an open-source and easy-to-build IoT module for micro-mobility platforms. This module would help set up, configure, maintain, and track micro-mobility vehicles from the cloud. During the initial requirements gathering stage, all the stakeholders involved agreed on a specific list of scooters our IoT module would support. We ordered actual scooters to our office to be able to run compatibility and functional testing of our PoC prototype. Finally, when cloud integration was implemented, we ran integration testing on the whole system which also included the client’s back-end and user apps for ordering scooter service.
To make sure we selected the most appropriate hardware, our team ordered several types of components from different suppliers. The goal was to find hardware that was both durable and widely available on the market. In the end, we settled for a Raspberry Pi Zero.
Lesson learned: You have to be persistent while aiming to create a durable device, especially when it comes to selecting hardware components. In the most complicated cases, hardware testing services provided by another company can come in handy to ensure your product’s quality.
Firmware testing
What if you miss a critical bug in your device's system? What mistakes will your end-users face when using it? Maybe your product won't connect to Wi-Fi or will reboot when it's not supposed to. To make sure none of that happens after the product is shipped, your team should perform thorough firmware testing.
Carepath: Healthcare device for remote patient monitoring
Carepath Technologies GmbH is a telemedicine platform that aims to monitor symptom-specific data in a home environment, aiding physicians with diagnoses and treatment of chronic disease. Hardware was one of the central pieces of our client’s platform. This is why our client gave it a lot of attention and asked to run not only functional testing of the device but also load and stress testing, performance testing, and penetration testing.
Using their experience from previous test cases, our QA engineers set up several boards with our firmware and remote connection to a testing cloud environment where they simulated high loads of data traffic and requests. On top of that, they measured how quickly the boards would heat up and perform a reboot at a certain temperature threshold to prevent overheating. We added security measures like code obfuscation to prevent cyber attacks on our hardware if, for example, somebody wanted to reverse-engineer our firmware binary, which had to be tested as well. In the end, all test cases had to be properly documented to pass the MDD/MDR certification process.
While testing OTA, firmware updates frequently broke Raspberry Pi OS. We created an operating system image and updated it directly from our office, which allowed us to ensure better and faster testing.
Lesson learned: Healthcare device testing can be a challenging task for tech companies. You need to have a proper data processing agreement, DPA, (GDPR compliant), with your engineering partner to ensure the protection of user data during the development and testing phases. Also, make sure to discuss and document as many test cases as possible to record all nuances that might be useful to your client in the future.
Want to learn more about building an AI-based healthcare platform for remote patient monitoring? Check out our webinar with Daniel Young, Managing Director and Co-Founder at Carepath Technologies:
Doppel for wearables
Doppel is a wrist wearable that uses controlled vibrations to help people feel either calmer and focused or more energized and alerted. We built a mobile app that helps find defects in assembled devices at the production factory. In its core, it's an Android app that factory workers run with a connected over BLE device. It can test the most important software features to ensure full working functionality. To check whether the device could properly process our requests and receive the correct data packet, we used Nordic’s nRF Connect application which went through the code and searched for errors. We compared the device’s response with the data specified in the documentation. For example, using the Doppel app, we commanded the device to turn on a specific rhythm of vibrations and checked via nRF Connect how the wearable device responded. Our team also tested Bluetooth connection, since if the device lost connection with the mobile app, the system would have to regularly re-establish it.
Lesson learned: Risk prediction will help you minimize the possibility of defects and avoid costly product returns. When in the manufacturing stage, think about building automated quality control software to avoid bugs. You should check how the device works when the connection is lost so that the end user would not struggle with reconnecting your product with the app.
Top tips for hardware and firmware testing
IoT systems and hardware testing plays an essential role in the product’s success. Each product requires a different testing approach. You can predict whether the technology will meet industry standards and user requirements with the help of different testing approaches and types. Here are some of our recommendations backed by plenty of trial and error:
- Plan the testing stage of your product in advance
- Don’t neglect issues found during the development process
- Consider multiple approaches to problem solving
- Team up with a reliable partner who is experienced in testing
Ensure your product's quality with Lemberg Solutions
We do our best to consider all critical incidents and cover the greatest possible number of issues during initial preparation and the development stages. Our skillful QA team will beat the bugs out of your device in numerous ways to find all its vulnerabilities and remove them. Our engineers hold ISTQB and ASTQB certificates which indicate their advanced professional level.
For automation testing, we use the following tools: Selenium WebDriver + PyTest and Behat (Web UI), Apache JMeter (Web Performance), Robot Framework (IoT), Appium + PyTest (Mobile), Postman and PyTest (API), and HTML_CodeSniffer (Accessibility). For manual testing, our team applies Charles Web Debugging Proxy, WEB Browser Developer Tools, Postman, DBeaver, Robo 3T, and Lighthouse (Web), SSH, MQTT Explorer, Azure Storage Explorer, Tera Term or Termite, ST-Link Utility, Nordic nRF Connect (IoT), Firebase Crashlytics, DB Browser for SQLite, iTools, 3uTools, Xcode, Android Device Monitor (Logcat), Droid Explorer, Android Studio, Scrcpy (Mobile). When testing hardware performance, Lemberg engineers perform smoke testing, functional testing, UI testing, automated UI test scenarios development, and defect tracking in Jira/Codebase/etc.
With Lemberg, you can gain a much better sense of control over your budget since thanks to our experience in embedded development services and IoT development services, we can anticipate numerous nuances that might arise when the product gets to the end user. This is how we make sure you avoid unnecessary expenses after the product’s rollout.
If you are not sure which testing scenario is suitable for your product, or simply wish to check how your device will perform under different conditions, get in touch with Slavic Voitovych, our Head of IoT Business Development. He’ll tell you everything there is to know about our expertise in delivering and testing IoT products.
