Coming soon… Free open source Selenium power tool!

We’ve been working feverishly and, frankly, ignoring social media to focus on developing cool stuff.

At the moment, we’re working on open sourcing one of our creations–a Java-based Selenium super-driver. We should have something out within the month.

The BrowserUp SuperDriver simplifies the creation of clean firefox and chrome profiles for headless selenium web browsing. It should largely be a drop-in replacement for the standard Selenium Drivers.

But wait, there’s more! Performance assertions, databanking, networking assertions… and more! Join our mailing list and be the first to try it!

Hello world!

This is our first blog post. Feels good!

Who are we?

We are 3 developers working on an “opinionated” performance testing tool. We have more than 45 years of development experience between us and experience building the previous generation of load test tools. You might have used things some of us built already. We also have experience building and scaling the websites of large, publicly traded companies.

We think testing is hard to get right. We’re making tools to help you get it right. Come along on the ride with us!

If you’d like to be on our beta early-access list, join our mailing list.

Types of Web Application Load Tests

There are a number of different common types of load tests. They each serve different purposes, and vet different capabilities.

Real-World Concurrency Test

A Real-World Concurrency test simulates normal production-like usage and determines the application’s scalability. This test is probably the single most important test for vetting a site’s capabilities.

Purpose: Vets the application for a certain number of users with production-like traffic and verifies performance characteristics.

Detects: Bottlenecks, issues including inadequate web server connections, CPU, memory, keepalives, inefficient queries and algorithms, load-balancer issues, worker thread/process issues, database connections, query problems and more.

Stress Test

A Stress Test verifies the websites capabilities when pushed passed expected capacity.

Purpose: Determine upper limit of system capacity and the next expected bottleneck.

Detects: Website breaking point, website degradation profile under increasing load.

Failover Stress Test

A Failover Stress Test verfies website performance when resources are taken away and recovered.

Purpose: Determine how failover conditions are handled including reduced web server, app server and database offlining, as well as recovery from changes in these conditions.

Detects: Correct behavior in the face of reduced capabilities, load balancing, automated recovery, error handling.

Endurance Test

An Endurance Test, also known as a soak test, involves testing a system with typical production load for a longer period of use.

Purpose: Verify the application can handle extended usage without resource scarcity issues.

Detects: Resource utilization problems over time where usage of a finite resources is exhausted over time. Typical problems include memory leaks, file handle leaks, database connection recycling, unrolled log files absorbing all disk space.

Bandwidth Test

A Bandwidth Test verifies the amount of bandwidth that your web server tier can pass. This test uses a large image hosted on your web server (not a CDN!) to determine the throughput your website can push through. A Bandwidth Test verifies the promised bandwidth can be delivered given your current configuration. By running an isolated bandwidth test, other tests may be safely run without downloading assets like images so long as the implied bandwidth requirements of those tests is within your established bandwidth-ceiling.

Purpose: Verifies your web server and networking capabilities in isolation.

Detects: Limitations with web hosting providers, network misconfiguration, load balancers, web server, and connection problems.

Baseline Test

A Baseline Test verifies website performance under a minimal-traffic scenario.

Purpose: This test determines best-case performance measurements for page load times.

Detects: Best-case page load time. Issues in testing scripts themselves.

Home Page Test

A home page test is a simple test that can verify some basic scalability issues.

Purpose: For static websites, this single-page test provides a decent estimate of a website’s capabilities. For true web applications, a home page test provides fast feedback on some problems.

Detects: Issues on a website’s homepage, as well as common web server issues and some configuration issues.

Announcing Our Load Test Scripting Series

We’re starting a Load Test Scripting series where we’ll cover common areas of site functionality, how to properly script them, and considerations for each type of script. We’ll be focusing on common functionality like authentication, product search, purchase, chat, and more. The posts will be tool-agnostic and focus on doing things right rather than any particular vendors or tools.

Our first post on testing your site’s authentication (login/logout capacity) is up already. We’ll add a link to each area below as we post them.

Scripting for Load Tests

Keep an eye out for more from us–join our mailing list or subscribe to our RSS feed to keep in touch.

Website Testing QA Link Roundup

Here is a roundup of a few QA testing-related links we’ve picked up in our travels.

40+ Free software testing e-books

Big list of naughty strings for testing input data

Google Testing Blog: Just say No to more end-to-end Tests

Top Browser Extensions For Testing Software – Ministry of Testing

Is QA Dead? — ThoughtWorks Featured Insights

For REST requests and #testing, we love with Postman. Especially the sharing features.

Enjoy!

Estimating Concurrent Users for Load Testing With Google Analytics

When load testing, one of the first questions to figure out is how many concurrent users your testing should target for your real-world concurrency tests. Developing an estimate of your site’s maximum concurrent users provides a starting point for defining future performance goals. For a new site, this can be a challenge, but for an existing site, there are many sources of data you can work from to understand your site’s performance target.

If you know how to parse server logs, or have access to someone who can, the most accurate way to determine out your maximum concurrent users is probably to parse the log entries or use a log viewer to figure this out. You can also use Splunk to calculate this number if you’re a Splunk user. Here’s an example.

But, often times a ballpark number is good enough. For that, Google Analytics works just fine.

Peak Concurrent Users

First, let’s look at your peak concurrent users. If you are using Google Analytics, jot down your average session duration. This thumbnail graph can be found on the main dashboard.

Screen Shot 2016-02-01 at 10.17.04 PM

Next, select a full year in the date-selector at the top. Find your heaviest during the year. Select that day and look at the hourly numbers and find the hour with the heaviest traffic and jot down how many users that is.

peakgraph

Session Windows in an Hour

Let’s figure out the proportion of sessions per hour. Assuming a 6 minute session length, this looks like this:

(60 minutes / 6 minute average session length) = 10 sessions per hour

Concurrency

Now let’s take the traffic for our busiest hour and work out how much of that falls into each session-sized window.

4000 hourly users / 10 sessions per hour = 400 concurrent users

That’s pretty good, but those users are not evenly distributed across the busiest hour. We’re really planning for the busiest minute in the hour, because if we undershoot and can’t handle the full weight of the traffic, people arriving during peak traffic could be deprived of our website!

Account for Some Variance

So we need to make allowance for the uneven-ness of traffic during that hour. For our fudge-factor, we’ll say 2x to account for the uneven spread. Based on what we usually see, 1.5x is typical, but 2x is safer. So 1/10 of the hourly load is 400, and we’ll use 800 as our number to ensure we’ve accounted for variance and we know we’re covered.

This concurrent users number provides a starting point from which to reason about your performance goals.

Future-proofing

In physical engineering, engineers plan for 3x the listed max capacity. So an elevator expected to carry a maximum of 3,000 pounds will be tested to 9,000 pounds. If you are growing, 3x or greater might make sense. So remember that this base number is a starting point, and we should scale our goals up or down, as appropriate to the business objectives, targets, and SLA for your website.

 

Real Browser Testing (Selenium) Link Round-up

Here are some interesting links we found on real browser automated testing:

Having Selenium wait for a page to load after a click can be tricky. Here’s a good way to do it.

PhantomJS vs. Selenium comparison by Phantom’s author.

7 things you need to know to make Selenium tests reliable and maintainable.

Write up of page objects pattern in Selenium.

Automating an HTML 5 Range element in Selenium. Is this the best we can do?

Best Free Selenium training tutorials. 

How to Use TestNG Annotations in Selenium (with Examples) 

 

browser-logos-small

Web Browser Market Share Trends

Some trends stick out when reviewing web browser market share trends for 2015. Internet explorer lost market share, and Chrome gained almost the exact same ground.

  • Internet Explorer market share fell by 8%
  • Chrome market share rose by 8%
  • Firefox held steady around 12%
  • Safari lost around 0.75%

Having spent many, many man-hours on IE-specific hacks, developers are surely crying crocodile tears over this tragic loss of Internet Explorer market share.

Source: NETMARKETSHARE

Reuse your functional tests for load testing