Who’s Using Your App? Examine Logs for Testing Insight

When testing an application, have you ever thought to yourself, "I wonder who uses this"?

I've been curious about that myself. In a previous role, I worked on a project management application for accountants. As someone who's not an accountant, I didn't have any experience or even business knowledge of the product. How accountants used the product on a daily basis was a mystery to me.

There are several ways testers can find out who is using an application and how. One approach that is becoming more commonplace is analyzing the application’s production logs.

Logs are lists of lines of text output from an application in a given environment, such as a test server or production server. They can be helpful for testing purposes because they provide real feedback and insight into an application as it’s being used, as well as information that describes or can even help solve bugs.

Each line of a log corresponds to some event or occurrence in the application. A log line could be informational ("A user successfully logged in at "1:00 PM EST"), a warning ("The current number of users is 90 percent of the total allowed concurrent users"), or an error ("A valid user login failed unexpectedly"). Log entries can be output from the application itself ("The number of logged-in users at a given time has reached a hard-coded limit") or from the application's environment or the system running the application ("The server has run out of memory and cannot allow any more users to log in"). Most logging systems provide a timestamp for each log entry, often to the millisecond, and each log entry follows some standard format. This can provide useful insight into the question "Who's using this application?"

An Example of an App’s Log

As an example, here's a simple application called Corsica. It generates random numbers based on an HTTP request and provides these values in a response.

Let's say this application gets two requests, one valid and one invalid. If we take a look at the logs provided in the application infrastructure, we see the following lines:

The two lines correspond to each request. Each line has the same format, starting with a timestamp of when the request was sent. The valid request returned an HTTP status of 200, while the invalid request returned an HTTP status of 404. You can also see information such as the fact that each request originated from a Chrome version 57 browser and what the end of the HTTP request URL looked like. This log contains other lines as other requests are made, each of which has data similar to these two highlighted lines.

In the case of our sample application, logs are generated automatically as part of an application hosting service (Google App Engine) and can be accessed by members of the project. This is true for many hosted web applications, but often, logs are generated and kept internally on a team.

So, back to our question: Who uses this application? Here are two approaches we can take to find answers.

Answering the Question Qualitatively

One approach to answering the question of "Who uses this application?" is to look at log data overall and find possible patterns.

One of the easiest pieces of information to analyze is how large a log is. If an application's log can fit on one or two pages in a word processor, chances are the application is being lightly used. If an application's log cannot be easily stored on a single hard drive, chances are the application is being heavily used. Seeing how much an application is being used can be an insightful, if surprising, way to find out how active users are on a given application.

An application's log size can also be an indication of an its age. It's probably unlikely an application with one million log entries was written an hour ago, just as an application with two log entries is unlikely to have been written twenty years ago.

Another easy way to get an idea of how an application is being used is to check how quickly log entries are being added. This gives some indication of how many users are currently using an application. A log may be large, but if no entries have been added in the past week, the application is not too active now. There may also be patterns in log entry speed. Some applications may have more entries added during an eight-hour period that corresponds to working hours than outside this window, providing some insight into when users are actively using an application.

After looking at these relatively easy first impressions of a log, you should take an exploratory approach. It’s a good idea to dive in and see if you can find any patterns. You could take a set number of entries that are most recent or from a given period of time and see if anything jumps out. Are most log entries errors, warnings, or something else? Do most log entries look the same? Are there any particular messages that appear more or less often? Are log messages easily understandable? This may be a good opportunity to apply blink testing. Quickly analyzing the data could provide a basis for further investigation.

Answering the Question Quantitatively

Looking at logs from a more quantitative view can also provide great insights. Here, we're looking for hard numbers instead of comparisons.

A tester may want to know precisely how many errors were logged in a particular time frame. Finding three errors logged in the past week provides a different state of affairs from having six thousand logged in the past week. For sufficiently small logs or logs that grow slowly, you can do some quantitative analysis by hand, but most of the time this is impractical. You'll have to look at the entire log to get good quantitative data.

For many testers, this approach may seem daunting. How would you get data like that? Remember that log entries are typically standardized. This means logs are a great candidate for parsing using scripts. You could use an off-the-shelf parser or write your own. Some modern logging systems have built-in tools that developers can use, so it's worth talking to the individuals responsible for logging to see what data you can pull out.

Log parsers can generate data sets that give you specific insight into how users are using an application. In the past, I've seen data presented like this from production logs. The most common API call made was the login() call, which makes some sense, given that the application was a proprietary web application used in large companies. All users would have to log into the application before doing anything, so this is a helpful check on our testing assumptions. This data was presented in graph form, with various charts representing the number of calls being made to some server-side APIs of interest. This was really helpful in figuring out what APIs were being used the most and the least, which in turn gave some indication of which parts of the application users are using the most and the least.

Use Logs to Inform Your Testing

There are many approaches to answering the question "Who uses this application?" The main advantage to analyzing logs is that testers get real user data. Instead of guessing or hoping they know how a user uses an application, logs show which parts of the application are getting used and what's going on with the application in production. Testers can use this data to inform other testing practices too, such as generating better test data or creating new exploratory test themes.

Most of all, logs may give testers some assurance that someone is actually using the application or feature they're testing.