Critical Analysis Of KDT — a reply to Nikolay Advolodkin
A few days ago, Nikolay Advolodkin posted on his linkedin page that “relic of the past. Something that we used to do when we didn’t have the right tools for the job. In today’s world, it’s no longer required. Removing it from our toolbox will help us to succeed with UI test” and he linked an update article from his blog about the problems with KDT.
This is my reply to his arguments.
Introduction
First and for most! I think Nikolay has been doing an honorable job listing automation anti-patterns and best practices, it is a great piece of work that he has been maintaining and revising seemingly for 3 years already.
He maid several points, some of them I agree with, and I believe that following these practices will help you build great automated tests.
I also think it is good to criticize various methodologies, this is how we make progress.
However, on that particular topic of KDT, I’m afraid that Nikolay totally mis-states his case.
I think he misrepresents what KDT is really about and his conclusions don’t follow from his arguments.
I’ll give an answer to his arguments point by point, but before then, I would give a short description of what KDT really is and what’s the purpose behind it.
What is KDT?
Keyword Driven Testing (KDT) is a methodology in software testing where in the prescription of test cases is defined by a sequence of actions (otherwise known as keywords).
So right from the get go — KDT is not exclusive to UI testing, but can cover multiple domains of testing from API to database or IoT.
The main advantages of KDT is the ability to easily maintain test code as it separates the infrastructure concerns from the test scripting.
The tests are written in a straightforward sequence of actions that are highly readable, as these are almost written in plain English.
One famous framework for KDT is Robotframework which I covered in a previous post.
With it’s tubular syntax, the test scripts are easily understood even by non coding professionals such as sales personnel, product managers, IT, subject meter experts (in my experience, Biologists and Veterinarians) and even costumers.
This readable syntax allows automation developers to delegate the writing of the test scripts to manual QA professionals and I’ll discuss about that later on.
The cons usually associated is a more difficult leaning curve when compared with capture and replay methods, however it has been well established that capture and replay methodology becomes very costly in the long run.
Nikolays Critics and the reply
Nikolay refers to KDT as being an “anti-pattern”.
Anti-pattern is a term used in programming to describe common practices that is ineffective and can lead astray.
To make his case he listed 5 reasons why he thinks KDT is a problem.
I’ll answer each on of these chapters including the prolog in the following sections.
0: non-squitter
In the prolog of this section, Nikolay argues that “Keyword Driven Testing (KDT) is a remnant from the early to mid-2000s when QTP aka UFT was popular.”.
This is a dead give away, he’s referring to old practices that were good enough at the time (some 15–20 years ago).
How’s that makes KDT an anti-pattern?
It would be the same as criticizing Selenium version 1- which was developed in 2004, the early mid 2000s which Nikolay refers to, and then to conclude that selenium is an anti pattern.
Then he talks about writing your keywords sequence in an excel sheet which he’s right, it was popular during the early 2000s because there weren’t as many alternatives as we have today.
1: “Almost nobody in the world is doing KDT any more”
I think Nikolay defeats his own goal in this sentence.
If no one uses KDT then what’s the point debunking it?
But of course, he’s not entirely wrong.
Nobody uses KDT in the way that HE prescribe it, but KDT has changed vastly over the last 15 years of open-source development.
He goes on to define his successful criteria of testing, the first one being “100% reliability from test automation”, which is totally vague and unclear, but then he argues that “In my entire career, I have never seen a successful implementation of Keyword Driven Testing.”
Well of course!
You work as a test automation consultant, if your clients were already successful in building automation infrastructure all by themselves, why would they reach out to you in the first place? this is a classic case of selection bias.
Besides that, it stands to reason that as an individual who has a pre-assumed belief that KDT is an anti pattern, Nikolay has a confirmation bias in which he tends to remember the bad implementations of KDT while leaving out the more successful one.
Now just to clarify, I’m not trying to be harsh on Nikolay, we all have our biases and presuppositions, we are humans.
Regardless of cognitive biases, I don’t think that anecdotal testimonies counts as evidence, either in favor or against KDT.
He goes on to say “Even worse, no automation luminary (Simon Stewart, Titus Fortner, Angie Jones, Alan Richardson, Paul Merrill…) even mentions KDT when they talk UI test automation…”
With all due respect to these awesome individuals, whom I hold in the highest degree, the fact they seemingly never addressed KDT doesn’t make it an anti-pattern, this is the argument from authority fallacy.
2: “Unnecessary code must be written to read some external source like Excel”
Sure, if you write the entire test automation framework by your self, you’re going to write the code for it — test automation framework doesn’t fall of from the moon…
similarly, if you developed a proprietary protocol you would have to write the code to implement it, if you created a proprietary database engine you would have to write the code for it…
But the question is why would you do that?
There are already products out there in the market, either commercial or open source that already do all that for you.
Robotframework is an awesome framework with reach eco-system where you don’t need to write any code to implement KDT.
In addition to test automation it has been successful as Robotic Process Automation platform.
He give an old example of code that he implemented back when he was a junior developer that parses an excel file.
And this is evidence that to do KDT you need to write a lot of code…
He then contrast it with a better code of data driven test using fixtures.
But this is again a non-squitter.
Keyword driven testing and data driven testing are not mutually exclusive.
You can have a keyword driven test suite that is also data driven, like this .
Now to his credit he does mention that “honestly, it’s not a 100% apples to apple comparison, but it’s close enough”.
but that is an understatement…
This is not apples to oranges, this is plastic apples with organic oranges, it’s not “close enough”.
3: “More complexity by introducing external teams”
Whether adding more teams is better or worse, this is really not part of the discussion, you can collaborate with other teams in any paradigm of automation testing and it can be successful or a failure.
I think the interpersonal relationship would account for much more of the success (or failure) of the collaboration then the actual framework.
But in any case it is irrelevant.
4: KDT forces you to create your test runners
Again, the same as #2 — why would you develop proprietary testing framework if it has already been developed for you?
Use robotframework, it already has the runner, reporter and everything that you need to run you test suites.
5: “KDT forces your tests to be written with interaction commands in mind”
This is simply NOT true.
In robotframework and in any KDT framework you can wrap what Nikolay refers to as “element interactions” into domain specific keywords which he refers to as “user actions”.
Nikolay then refers to a talk by Titus Fortner where in he differentiates between imperative test scripts and declarative test scripts, see in the picture below:
THEY ARE BOTH IMPERATIVE!
Just because you wrapped some parts of your code into a method and reused that method in several places doesn’t automatically make your code declarative!
imperative and declarative has nothing to do with how many wrapping functions (or methods) your code contains.
It has more to do with the way our program manages state.
In imperative style, we change state by a sequence of statements, and in declarative style we express high level logic without flow control.
The best example for declarative programming is SQL, in which we do not mutate internal state but rather declare logical expressions.
So wrapping up your “low-level” actions into high level domain specific functions doesn’t make your code declarative and it is not bared from KDT methodology in anyway.
On a final note
It seems like both the article and the videos linked are influenced by the page object pattern as being the good way to code automated tests.
As I argued in my 2 part article, page object is overrated.
It’s alleged benefits can easily be achieved by procedural style programming ,which KDT ascribes to, without the overhead of conceptualizing all your actions as “doers” and shoving every aspect of your code into a page object class.
Most of the advice provides in the article are sound and reasonable, but this particular chapter seems to be inaccurate and not up to date with the last decade or more of KDT practices.
