Sometimes, even knowing what is possible can be big step in the right direction.

In the past I’ve written about impossibility theorems in mathematics. These are class of theorems that definitively conclude that, given a set of constraints and conditions, something is impossible. A straightforward example is the impossibility of tiling the plane with regular pentagons. Try finding some regular pentagon shapes and making them tile a floor; you will find you cannot do this.

On the flip side to this, there is another class of theorems in mathematics called existence theorems. These are theorems that show that, definitively, something exists given some assumptions. Existence theorems may not provide any information on how to figure out how to find that thing that exists, or even provide a clue to what a value is. They may not even guarantee there’s only one of the thing that exists. The only information an existence theorem (in general) provides is that a thing exists. One example is the contraction mapping theorem; basically if a function is a contraction - ie loosely speaking, the function’s input maps to a subset of the input, “contracting” the input to a smaller set - then iteratively applying the function will produce at least one “fixed point” or a single point the iteration converges to.

So what does existence and impossibility theorems have to do with, well, anything other than higher mathematics?

Human beings tend to like certainty. We like knowing certain things will happen or won’t happen, and avoid situations with uncertainty. Impossibility theorems and existence theorems are ways to figure out that, yes, something is certain. Whether that thing is certainly impossible or certainly possible may help us reckon with what to do next. This certainty may provide clues to how to find a solution, or to stop looking for one, or even to reformulate our assumptions.

In any case, knowing what is possible can be a big help, at least on a psychological level.

As software companies grow so do their engineering teams. A role that comes along with growing teams is the software engineer in test (SEIT), a software developer who primarily works with test automation and related testing infrastructure. SEITs can be extremely valuable to engineering teams as they can enable good automation practices for shipping quality code faster, and they can be a great touchpoint for test engineers and operations engineers to work with development teams. They can also be a bit tricky to hire since SEITs don’t have quite the same skills as application developers or test engineers.

Let’s take a look at how your might approach hiring SEITs at different levels.

Junior/Intern

I’ve worked with several interns (in Ontario these people are called co-op students) and working with test automation is a great introduction to being a software engineer of any kind.

When to hire

Juniors are great on teams with established SEITs and test engineers. In some organizations there are entire teams dedicated to test automation efforts. If you need someone to bang out many test code or take partial ownership of a large test automation project, a junior hire could be a great choice.

Like in other software develoment roles, hiring a junior usually comes with an expectation that the junior candidate will receive some menitorship and early career training and experience.

What to look for

Candidates for a junior SEIT should show characteristics of a junior test engineer and/or application developer. On the testing side, an ideal candidate should show curiosity about software or systems, the ability to develop at least basic mental models of how a piece of software might work, and good communication skills, written or oral. On the development side, an ideal candidate should show at least a basic aptitude for programming from school or otherwise. They should be able to read, write and understand code in some programming language even a language that your organization doesn’t use. Previous experience working in software development is nice but not a requirement. It can be safe to assume that junior candidates have worked with at most one test automation tool and likely none.

Key questions to ask

• “How would you test a pen?”: a classic test engineering question to access a candidate’s reasoning skills, communication skills and question asking ability.
• “Write a function to reverse a list in a given language”: a basic but straightforward assessment of programming skills. The language can be set by the interviewer or chosen by the candidate.
• “What is something you achieved that you are proud of?”: a question to get some insight into the candidate’s motivations and interests.

Intermediate

Teams getting started with test automation or parts of test automation could benefit from hiring experienced SEITs. Experienced SEITs can write test code but also set up needed infrastructure and coordinate with developers, operation engineers, and testers. Intermediate SEITs may also be able to start test automation efforts for new projects as needed.

When to hire

The main reason to look for an intermediate SEIT is when automation efforts exist but have grown past being managed part time by single individuals. For example, a team may have written some Selenium-based tests to test some scenarios, and at first a small number of tests can be handled by the team or specific members of the team without a test automation speciality. Once the number of tests grow to a certain size and/or complexity, that team may want to have a dedicated person looking after them. This could also be the case with automated performance tests, service-level tests, and so on. Or a team may seek to level up on test automation skills.

What to look for

Test automation is a speciality of software development, and as I’ve said before (if slightly reworded) SEITs are developers. Evaluating them as you would other intermediate developers isn’t unreasonable. Look for one or more roles where they worked as SEITs, DevOps engineers, or other forms of automation. Note however that SEITs may have written many, many lines of code but aren’t traditionally skilled in areas of programming. Most SEITs wouldn’t be able to code low level algorithms like linked lists or binary trees and may have never really used map/reduce/fold approaches. Intermediate SEITs should be able to discuss tools they’ve used well, and strong intermediates can identify differences between tools and levels of testing. And they can definitely write tests; any intermediate SEIT should be able to write a few tests for at least one layer of an application, such as unit or component tests, API level tests, and/or end-to-end tests.

Key questions to ask

• “Write three automated tests in a given language to test this class”: This question assess a candidate’s knowledge of test development code writng
• “Here is an example login test. How would you improve it?”: A possibly fun exercise that helps evaluate a candidate’s knowledge and technical communication skills.
• “Suppose a software bug is found using test code that you wrote. What would be your first reaction?”: Teams handle bug reporting in various ways with varying levels of success, so this question can shed some light on a candidate’s experiences.
• “Describe a time that you and a team mate - could be an app developer, tester or SEIT - disagreed. How did you resolve your differences?”: Classic interview question for verifying what a candidate’s personality is like.
• “What’s your favourite continuous integration tool and why”

Senior

If you work for a organization that has experienced growth and is now “established” in some sense, you may find some senior SEITs or other roles similar to that. In medium- to large-sized companies - companies a typical person may have heard of - there are often whole testing teams and sometimes engineering teams dedicated strictly to internal infrastructure. If you consider automated tests as infrastructure, this means you’ll need dedicated folks to oversee project health. This includes writing and maintaining test code, but also includes managing people and computational resources. Good SEITs know how to put things together from a testing standpoint, and how to organize accordingly.

When to hire

This is definitely a bit of a “I’ll know it when I see it” situation. Teams can often get pretty far with hiring and retaining good intermediate SEITs or a combination of SEITs, testers and DevOps folks. In many larger organizations, test frameworks can grow to require their own dedicated team leads and managers. This is often where senior SEITs come in. Another scenario is where test automation efforts expect to ramp up, and having someone knowledagble in test automation can guide the process.

What to look for

Senior SEITs have worked with multiple forms of test automation at most or all layers of an application. They will likely be well versed in more than one area of test automation, and have built test frameworks from stratch and worked with existing test frameworks. Depending on the circumstance, they also may have some team lead or engineering management experience, and so may be able to work with more junior members on a team in addition to coding and configuring infrastructure. One of the most valuable aspects that senior SEITs can bring to the table is an informed opinion on test automation topics.

Key questions to ask

• “Suppose you were asked to review a test suite of 1000 tests. How would you approach this?”: This question gets at how a candidate approaches a relatively complex situation. The idea here is spur discussion more than provide a flat answer.
• “What’s your preferred test framework and why? What’s your least preferred?”: In light of hiring someone who brings experience and opinions, this question asks for some of those opinions directly. It can also be a good indicator of communication abilities and temperament.
• “Describe a time that you and a team mate - could be an app developer, tester or SEIT - disagreed. How did you resolve your differences?”: Again, a classic interview question that could be critical if the candidate in question has to manage people.
• “Are you willing to mentor junior app developers and SEITs?”
• “Explain the value of end-to-end testing to me if I were the CEO of the company”: Test automation can seem like an esoteric speciality to those not in engineering organizations. Senior developers of all kinds (in my opinion) should be able to talk in business terms in addition to technical terms.

Happy testing!

This week I changed a toilet seat on my main floor toilet. This was a substantial achievement, and I was able to do this thanks to my extensive mathematical training.

If you’ve ever changed a toilet seat on a standard toilet (in North America, anyway), you’ll know it is fairly straightforward: undo the screws or latches on the old seat, remove the seat from the porcelain, remove any screws or nuts, and attach the new seat. I’ve done this a few times in my life and it is not too difficult.

However, this toilet seat was not easy to replace.

It was a style installed by the previous owners of my house that did not have easy to remove latches or nuts. The hinges were chrome and only attached by thick screws that went through the porcelain and had plastic nuts that held the screws and seat in place. The nuts were fastened extra tight, and could would not budge. It took me multiple attempts with multiple wrenches, and I could still not remove even one nut.

The turning point was realizing the entire problem (remove the old seat and install the new one) depended on solving one small aspect of the problem (remove one nut from a bolt). If I could solve this one small problem, I could solve the entire larger problem.

In mathematics sometimes a large problem can be reduced to a smaller problem or to a set of smaller problems. For example, properties of square matrices are often related, and knowing one aspect of a square matrix - for example, that it has a determinant of 4 - means you know lots about the matrix in question. With a determinant of 4 (aka not zero), we know the matrix has full rank and that it is invertible. Many problems in dynamical systems theory come down to knowing the eigenvalues of a square matrix, so even knowing what the determinant is can help provide information about the entire system.

In my case, I knew I had to remove the plastic nut. Using a hammer and chisel I hacked off the nut, which loosened the bolt enough that I could shift and remove the second nut on the other side, and in turn, remove the old seat. I was able to reduce my larger problem into a smaller one.

This shift in thinking also had a psychological benefit. All of a sudden a difficult multi-step problem became a focused single-step one. I moved the problem forward.

Math is surprisingly helpful to learn as a way to learn problem solving and thinking, even in situations that look like you’re going down the drain.

Friendly Announcement: If you liked this post or ones like it, sign up for my math newsletter

It’s been a heck of a few years for me, personally and professionally. One thing that I am grateful for is returning to Targeting Quality 2023, the annual conference held by KWSQA. This year the conference was held in Cambridge, Ontario. Here are my thoughts on it.

Overall

• The testing community is still a great community. Attendees were engaged, and even as someone who’s done primarily test development for his career I was still able to have great discussions with testers about all kinds of topics.

The Talks

• The first talk of the day on Presentation day that I saw was Bas Dijkstra’s How well do you wield your tools? - The importance of craftsmanship in test automation which was a part of the automation track. One of the key messages in this talk was that test automation can and should be treated like production software development, which is something I’ve been saying for a long time. It is good to hear that this message is still coming through strong.
• Test automation is definitely a mainstream topic in software development, but there’s still so much that can be done by application developers in terms of test automation. Concepts like property-based testing, contract testing and security testing are all highly relevant for test development.
• Fuzz testing: still not a common topic outside of the security engineering world
• Another talk that I enjoyed was The Automation Firehose by Thomas Haver of M&T Bank. This talk gave a great overview of how to think about growing and scaling test automation teams with a superhero theme.
• Ben Oconis’ talk on breaking down silos in organizations sparked some thoughts from me and folks online about the role of “test specialist” in software organizations. Ben’s talk was really good at outlining some organizational aspects to quality, which aren’t necessarily “testing” skills but important nonetheless.
• On this note, software testing is still a great place to learn about good team building practices. Lean Coffee, for example, is an excellent format for getting engaged folks to talk about a few different subjects in an effective (and time efficient!) manner. There were two sessions of lean coffee lead by Lisa Crispin and Janet Gregory. I attended one of these sessions, and I can’t recommend trying lean coffee enough.
• My favourite talk of the conference was the keynote, simply entitled STAIRS! by Jenny Bramble and Jenna Charlton. Entertaining and insightful, Jenny and Jenna show great examples of cognitive biases and how they can effect how products get designed using examples of stair designs (some of which were hilarious bad). This keynote showed that people in the testing community are digging into concepts such as design thinking and how that impacts testing work and quality, and doing so ahead of other in the software development world.

The Conference

• I have to say: this is one of the best run conferences I have attended. Targeting Quality was 2 days this year, one day of workshops (which I did not attend but heard good things about) and one day of presentations.
• The venue was well-prepared and suitable for the purpose, right down to stacks of pens and notepads available all over the place.
• Food and drink was great, including delicious Portugese-style custard tarts at breakfast (a local delicacy, in my opinion)
• The talks and schedule were well-organized and everything ran pretty much on time throughout the day. This is an underappreciated aspect of any conference, large or small.

Overall, this conference offered a lot of benefits for attendees and speakers. The hall-way track was lively, talks were great quality and speakers were well taken care of. The KWSQA board and Tina Fletcher should receive a round of applause for a great job.

Really, the title of the post says everything. Using a specific tool is not the task to complete. It is the tool.

This comes up often in software development, where folks like to discuss and plan around a tool instead of a task, but an analogy I’ve been thinking about is dishwashers.

I have a dishwasher and, by far, it’s my favourite kitchen appliance. Dirty dishes go in, I turn it on, and a little while later, clean dishes come out. It works, it works well, and it does what I need it to do. One day, I was thinking about the role of being a dishwasher in a restaurant, which is a traditional “entry-level” role in the hospitality industry. I wondered: Why don’t restaurants simply use one or more great dishwasher appliances instead of a full-time employee dishwasher?

This thought (which wasn’t too well-considered) lasted a few minutes before I realized the answer: there’s more to keeping dishes clean than washing them. Dishwasher employees need to clean guest flatware and tableware for sure, but also pots and pans and such for the back-of-the-house kitchen as well. These items need to be cleaned, and allocated back to where they should be, and this should be done in a timely way. Cleaning a dish is only one aspect of this job. The task is to keep items in a kitchen clean and ready as needed, where an automatic dishwasher appliance is a tool to help accomplish this task.

The same goes for test automation approaches, for example. Choosing Selenium over Playwright for browser testing is making a decision about a tool, but deciding what kinds of tests or aspects of a software development process should be automated is (likely) the task.

Keep this in mind when thinking about tools and tasks.

If you’re looking for a good time, check out the hot new tech subject, Cybersecurity.

(I’m channeling my best Stefon here.)

Seriously, this topic has everything. Two talks in this area from this year definitely have me thinking that when it comes to cybersecurity, the truth can be stranger than fiction.

Earlier this year, I attended a talk at BSides Rochester entitled “TikTok Under Attack: Attacker Uses a Popular TikTok Challenge to Lure Users Into Installing Malicious Package”. I was intrigued because I do enjoy TikTok, but was not prepared for the roller coaster ride of this talk.

The overall story goes like this: the speaker is a security researcher who was interested in starjacking, a potentially malicious process where a given GitHub repository is shown to have a much higher number of stars/stargazers than it actually does. This can be exploited by poor controls on package managers such as NPM or PyPI which don’t necessarily validate a GitHub repo’s metrics. Here’s a good intro to the topic.

This researcher was finding ways to understand how starjacking works, and found that there were malicious PyPI packages that if installed, will enable starjacking on particular repos. Installing a PyPI module does require some Python knowledge - it’s not quite as easy as clicking a link on a webpage - so the attackers had to find a way to get individuals to install this mysterious PyPI package.

This is where TikTok comes in.

The researcher found a challenge on TikTok called the Invisible Filter challenge. The idea was that TikTok created a filter that makes a person in a video “invisible” (you can only see a rough outline of a human while the shape blends into the background). The challenge was to create videos and have viewers guess who the invisible person is, what they’re wearing, etc. A variation of this challenge was for the person in the video to be naked so to “appear naked” on TikTok without actually appearing, er, naked. The attackers exploited this by posting links to their malicious PyPI package saying this Python package could remove the invisible filter showing the person in the video. This caused a bunch of folks (referred to as “creeps” by the speaker) to download and install the malicious package, thus facilitating starjacking.

That talk was a wild ride. Where it started and where it ended up were two different places.

I thought for sure this was a one-off style of presentation (PyPI! Starjacking! TikTok! Nude people!).

But wait, there’s more!

During this year’s BlackHat conference, a PhD student gave a talk called “Houston, We Have a Problem: Analyzing the Security of Low Earth Orbit Satellites”. Effectively, the researcher asked “How secure are in-orbit satellites?” and the answer is “not very”.

While I didn’t see this talk myself, I can imagine this having the same flow as the TikTok talk. For example, the researcher found that many security features of satellites are non-existent, likely based on the thinking that attackers on Earth couldn’t access them in any way. That turns out to be false, and even cloud services such as AWS offer Ground Service to low-orbit satellites. You can even get started for free.

Again, this talk had everything: basic hacking techniques, cloud providers, satellites…in space!

When I think of cybersecurity, I typically don’t think of talks that offer stranger than fiction style narrative, but it may be the best place to find some such tales.

Many years ago, I took two courses in graduate-level mathematics as part of a summer school program at Dalhousie University in Halifax, Nova Scotia. It was an enlightening experience that very much enjoyed, learning some math on Canada’s East coast (I highly recommend visiting Halifax in the summer).

In one my courses, we were given an assignment to prove a result from probability theory. The proof was relatively difficult but not impossible, and those in the class were not necessarily experienced in intermediate probability theory (myself included). The professor mentioned that there was a proof of this result published in at least one textbook, if not other papers and books. He also said he would accept a reference to this proof from a source, instead of proving this result directly ourselves. This was the first time I ever had a mathematics professor (or teacher of any kind!) tell me that I didn’t have to prove a result myself if I could produce a valid reference to a proof.

So I went to the main library on campus, did some digging, and found a proof in a textbook. I took down the reference, handed in my assignment, and received full marks for that section. Mission accomplished.

Engineers, software or otherwise, may have heard the phrase “a few hours of research can save a few weeks of development” and my experience with this proof was my first real experience with this line of thinking.

Sometimes there’s no reason to re-invent the wheel if you can find the work done somewhere else.

This is a part of a series of things developers should not build.

Often, developers need to write code that has some element of randomness. In game development actions may depend on a dice roll or coin flip. In scientific programming and simulation generating random fractional values within a range (between 0 and 1, for example) is a common task. From a design perspective, perhaps a developer needs to randomly select an image from a set of images to present in a front-end. In any case, a developer may think “I know, I’ll write some code myself to generate some random values”.

This is – with some narrow exceptions – not a good idea and a developer should not do this and instead use a pre-built pseudo-random number generator library. There are libraries for this task in most programming languages. Failing that, there are CLI utilities that can be called from the OS level.

Let’s take a look at why this is the case.

The Theory is Hard

First, what’s the difference between random and pseudo-random anyway? Isn’t this just a technicality?

The short answer is: no, it is not just a technicality.

Randomness is a difficult topic for human beings to understand. Truly random events have zero relationship with one another, meaning that one event has zero impact or influence on the next one. If I roll a six-sided die and it turns up a four, then rolling again has no impact on the next roll the next time if the rolling event is truly random.

Now consider about a computer program or script. It needs to run from the beginning to the end. Code runs in a predictable and deterministic way. By definition running a program from beginning to end cannot have any randomness. Computers can’t stop and roll a die.

Luckily, there’s a workaround: create a function for generating numbers that appear random, statistically. Such a function returns the same output for a given input. However, each generated number is “random-enough” to simulate a random event such as dice roll. Mathematicians have worked on creating algorithms and functions that output random-seeming values called pseudo-random numbers that have good properties for being close enough to random. These include algorithms such as linear congruential generators and the widely implemented Mersenne Twister algorithm.

(As a side note, if true randomness is a hard requirement in an application such actual lotteries using dice rolling, there are approaches that use physical phenomena to generate true random values. See RANDOM.org for more information.)

Given the theoretical issues, the creation of pseudo-random number generator (PRNG) algorithms provide a path forward. Sadly, PRNGs do not end the problems developers face.

The Practice is Hard

Implementing a PRNG seems straightforward. Find an algorithm, pick some values for various aspects of the implementation, and execute away. Many PRNG algorithms are effectively difference equations that start with a given value and iterate through a provided number of times. Easy right?

Well, no.

The first issue that arises is computation time. Generating a single value from a PRNG is fast, but what about generating 100 values, or 1000? In practice, applications require a random value on every command or execution which could mean that many, many random values need to be generated. If each generation requires many, many iterations of a particular algorithm this could lead to non-trivial execution times. In game development, this could be a deal breaker, since performance is crucial. Often PRNGs need to be fast at runtime.

Computer hardware has gotten fast thanks to Moore’s Law so even executing a large number of iterations within a function is pretty fast. However, speed isn’t the only consideration for PRNGs. There’s also the aspect of being “statistically random” which means choosing a “good” PRNG with good configuration.

An infamous example of choosing a “bad” PRNG algorithm was RANDU. RANDU was based on a linear congruence generator as mentioned above. While RANDU generated approximately random values, it had some obvious flaws. For example, RANDU only generated odd numbers. As well, after a while generated values would repeat, meaning there was a pattern that did not appear random at all. Worst of all, RANDU was taken an implemented in a variety of settings. Note that this was a vetted algorithm by smart folks.

While it is definitely possible to test a random-number generator for correctness and performance, PRNGs are sensitive to configuration and there are many pitfalls that a developer may encounter if they are not familiar with the problem space of numerical analysis and computational mathematics. Which in my experience is not a lot of developers.

However, smart folks have learned from their mistakes and now there are off-the-shelf PRNGs using vetted algorithms in most mainstream programming languages.

It’s all good, right?

Don’t Forget About Security

The last and possibly biggest issue with using PRNGs in code is security. If an attacker can determine the PRNG values being generated by a program, that attacker can exploit this knowledge. A helpful example is an online poker site. If each card that turns up is chosen using PRNG algorithm, then a player who knows what value the PRNG will output next can use that information to determine every single card dealt to every player. This knowledge can easily be exploited to manipulate each hand.

One of the reasons that RANDU was such a disaster was that it was not secure and it was widely used. Once an attacker know a piece of software is using RANDU they can exploit that information for other malicious purposes. Using a PRNG in a local toy piece of code may be completely fine but using an insecure PRNG in production could lead to major security holes.

Again this is not a scenario that many developers consider when deciding to use a bit of randomness in their code.

Mercifully, there are well-tested cryptographically secure PRNGs available in many programming languages.

In summation, choosing a PRNG built and maintained by experts is a better bet than rolling your own.

In the spirit of “You can’t buy integration” here are things that a programmer should never build:

• A Pseudo-Random Number Generator
• A Cryptographic Library
• A Numerical Integration Solver
• A Fuzzer

Each of the reasons why are straightforward: building each of these from scratch is loaded with pitfalls that could cause problems that far, far outweigh true benefit.

How you decide what events to go to?

This is a tricky question for most people, but even more so for people who go to events as part of their job. For example, developer relations advocates often present at conferences to tell people about their cool projects (like for example a AI-guided fuzzing tool). However, there are only so many hours in a day, and only so many events someone can attend in a year. How can you choose the events that work best for you?

One approach that I think works well is to take inspiration from one of the greatest actors of a generation: Christoper Plummer.

The late but great actor Christopher Plummer took every kind of acting role you could imagine: large scale blockbusters musicals to European dramas to Shakespearian plays (and even some lesser quality films). When asked how he chooses roles in an interview, Plummer said a role needs to have one of three characteristics: the role is interesting enough, pays well, or is in a great location.

This is how I think about events, and I think the Christoper Plummer rule works well. Stated for choosing developer events, you should consider going to an event if the event

• is interesting,
• pays well, or
• is in a great location

Interesting Role (or Event)

Developer events and conferences come in shapes and sizes and cover all kinds of topics, from the most narrow programming construct to the broadest area of industry. Depending on your interested or priorities, there are likely a few “major” events or conferences that take place on a regular basis. For the Python world there’s PyCon, for the infosec world there’s DEFCon, and so on. These events draw a lot of people and event simply showing up without a plan may still be beneficial. These major conferences and events are definitely something to consider, even if you’ve never attended or aren’t overly familiar with them.

Pays Well

Sometimes conferences will pay for speakers to attend, and in some cases this includes food, accommodation and travel expenses. Similarly, most conferences look for sponsorship from potential partners and vendors and in turn provide conferences passes. If you have the inclination to speak to people, from onstage or from a table, you may be able to attend a conference at very little out-of-pocket cost. Who doesn’t love free stuff? Finding opportunities to go to conferences at little to no personal cost might help make them more attractive to you.

An Interesting Location

Have you ever wanted to visit Disney? Then going to a conference or event in Anaheim, California or Orlando, Florida might provide the perfect reason to visit. Many places draw visitors for business or pleasure based on local culture, commerce or climate. If an event happens to take place in somewhere interesting to you personally, it might tip the scales in your favour of attending. A great location also makes a less valuable event seem a bit less bad.

If all the world’s a stage, then consider these three things when choosing where you’d like to be a part of the play.