Load testing Vs Functional testing— similarities and differences
Automation development is an umbrella term containing multiple disciplines for different purposes.
Test automation, performance testing, Robotic Process Automation (RPA), various aspects of Devops, deployment, monitoring and more, can all be conceptualized as fields of automation.
For us, automation developers, this is awesome!
Mostly because it means we have a lot of space for professional development with an ever growing size of unknown territory to explore.
It is to our benefit to understand the similarities and differences between the disciplines.
Not only it can help us perform our tasks in the most productive way, but also it can guide us in our career progression.
In this article I’ll review the similarities and differences between functional testing and Load testing.
This topic would be part of my upcoming talk in the LoadTestWorld conference, you’re all welcome to hear me talk about collaborative efforts in load testing.
The goal
Program testing can be used to show the presence of bugs, but never to show their absence!
— Edsger W. Dijkstra
In functional tests, our goal is to imitate a large number of user flows with a small number of concurrent users.
In load testing it’s the other way around, we imitate a small number of user flow with a large number of concurrent users.
Note that a “large” number of concurrent users doesn’t mean “as many as possible”.
Large scale capacity tests are needed, but for the most part our load tests will not include the absolute pick of our systems capacity.
Matrices
In functional tests the aren’t as many measurable outcomes.
We go over the defined user flows and add assertion statements to validate the behaviors.
In load tests on the other hand we are less interested in the behaviors.
Instead we put more focus on measurable outcomes such as error rates, free memory, diskspace, CPU utilization, latencies, reset counts etc.
once again, it doesn’t mean that there are no assertion statements in load tests, but compared with functional tests they are not as prevalent.
Ecological Validity
In empirical research, ecological validity refers to how well the experiment resemble to “real life” conditions, and the same concept should apply to software testing.
In the context of functional tests, our concerns are related to the user flows.
How realistic they are when compared to the users who consume the application.
For example, order of API calls or order of UI operations.
In Load testing we are more concerned with rates and load generation patterns.
For example, should the rate of request be constant in time, or should it follow some Gaussian patterns.
Test should be as resemble as possible to real life scenarios, otherwise we compromise confidence levels in our testing efforts.
Exploratory VS Robotic
Testing endeavors can be divided into robotic components; in which we perform repetitive scenarios that are prescripted with known expected outcomes, and exploratory components; in which (as the name implies) we explore our application in an unstructured manner.
In functional testing, the exploratory testing focuses on exploring user flows, discover new flows, discover bugs, discover inconveniencies or accessibility issues.
Robotic components, having being discovered in the exploratory phase, are then converted into automated tests that are done repeatedly in any of our work pipelines.
In load testing on the other hand, the exploratory component is more profound.
In load testing we tend to hit the system with loads and explore how well our application keep up, in other words, we are monitoring either the metrices of our application, the logs of our application, or even directly performing functional tests elements on the application while the load is running.
The robotic component of load testing involves execution of load test scripts on triggers (scheduled, pull-requests commit etc) with a smaller scale than we usually do in capacity testing.
Costs
When compared with load testing, functional test has a significantly lower costs, especially when compared to the big scale load testing.
The impact of high costs effects the frequency of our test executions, the more costly they are the less frequent they will be executed.
Moreover, some load test scenarios require longer time to execute (soak testing, volume testing).
So not only our ‘per minute’ costs are higher, but we are using more minutes of execution.
Root cause analysis
Root cause analysis in load testing is around finding bottle necks in our application or infrastructure.
In functional test we are going to investigate the behavior of our application under specified conditions.
Since “failure” in load test is more difficult to reproduce (owing to the costs and time involved), it stands to reason that root cause analysis is more difficult in load testing when compared to functional testing and solutions are harder to develop.
It has also been in my experience that with functional testing, it is more frequent that bugs are being put on “hold” for several iterations, but it is less the case with load testing, as performance issues have a severely negative impact on user experience.
Load testing root cause analysis can involve more 3rd parties (cloud services, storage services, load balancers etc) when compared to functional testing
Interpersonal relationship
Interpersonal relationship is crucial to both endeavors, however, I would argue that the impact of interpersonal relationship is more profound in load testing when compared with functional testing.
The nature of load testing, being more subjected to uncertainty and require more efforts to investigate root cause etc, require higher levels of team cohesion and professional maturity where all stakeholders are aware of the big picture in terms of project requirements and goals.
Ego must be set a side at any given moment, everyone must be engage in debates with an interest to extrapolate the unknown rather then to “be right” all the time.
Our human fallibility and cognitive biases can lead us in wrong directions and the only way to balance that is to create a good working environment where everyone is encouraged to express their thoughts and ideas and their peers will be willing to change their minds in according with the evidence.
Empathy, especially the cognitive aspects of it, ie perspective taking and theory of mind, is crucial to such an effort.
Conclusion
The differences between load testing and functional testing mostly revolves around the differences between quantitative analysis (mostly related to load testing) and qualitative analysis (mostly related to functional testing).
The similarities are more related to the common challenges in judgement under conditions of uncertainty.
Are there any other points of similarities or differences between the two you’ve experienced?
Please write them down in the comments bellow.
