Vegta For Performance Testing

Vegeta is a versatile HTTP load testing tool built out of a need to drill HTTP services with a constant request rate. It can be used both as a command line utility and a library.

Repo:  https://github.com/tsenart/vegeta

How to do load testing

There are a lot of different tools to help with load testing. The way they typically work is you construct an HTTP request, like GET http://www.example.com/some/path and tell the tool to send 1000 requests per second to it for 60 seconds. Usually you’ll get some form of report — typically in plain-text or graphical form which you can interpret.

The tools that work this way are too numerous to list, but some popular ones are Apache JMeter, Apache Bench (ab), Siege, and cloud-based tools like Loader.io.

The problem with basic load testing

In complex systems often there is often some kind of caching involved when processing a HTTP request. For example perhaps you have an API which takes some data, performs an expensive (slow) operation on it, and then returns the result. More often than not you can store the result in a cache, and just return that if the same request is repeated.

This poses a problem for load testing because typically load testing tests the same URL and HTTP body content repeatedly. Effort that is cached either by your API service or other services it depends on will be served faster than effort that is not, and so you’ll see a bias toward faster response times. This renders load testing results largely useless because the test data are not anything like real-world traffic.

To avoid biases like this there are a few options — examples are:

• Replay production traffic to simulate load
• Generate test traffic that is more like the real-world

Depending on the sensitivity of your data, infrastructure, and type of work your API does, replaying production traffic may or may not be an option. So in this article I’ll focus on generating test traffic.

To learn how to generate and load test with test traffic we’re going to:

1. Introduce Vegeta
2. Install Vegeta and see some basic use cases
3. Learn how to load test multiple targets
4. Generate some test traffic for a fantasy web service
5. Join our generator program with Vegeta

We’ll be writing a small program in Go, so if you want to follow along you should install Go from here if you haven’t already.

Introducing Vegeta

Vegeta is a command-line load testing tool (like ab and siege) written in the Go language. We’re going to use Vegeta because it has a very simple interface, and it supports multiple request targets and request bodies via a targets file or piped from STDIN.

We can make use of Vegeta’s ability to send traffic to multiple targets with multiple HTTP request bodies via a simple command-line interface to easily send a lot of varied, more ‘real world-like’ traffic to an API.

Installing Vegeta

Precompiled executables for Vegeta can be found here. Or, if you’re on a Mac and have homebrew installed, you can use the following command:

$ brew update && brew install vegeta

Basic use

The most basic use of Vegeta is something like the following:

$ echo "GET http://example.com/some/page" | vegeta attack -rate=10 -duration=30s | vegeta report

This will make an HTTP GET request to the URL given at a rate of 10 requests per second for 30 seconds, generate a binary report and then display it via the vegeta report command.

Testing multiple targets

Vegeta can also accept a list of targets, as a line-separated file or from STDIN.

An example (from the Vegata docs at https://github.com/tsenart/vegeta) goes like this:

In targets.txt you have:

POST http://goku:9090/things  
@/path/to/newthing.jsonPATCH http://goku:9090/thing/71988591  
@/path/to/thing-71988591.json

At /path/to/newthing.json and /path/to/thing-71988591.json you have two different request bodies in JSON encoding.

The targets.txt file contains targets separated by double-newlines, followed by a path to a file containing a request body. vegeta attack will automatically load the body content from these files for us at runtime.

Then you can invoke Vegeta to make the requests to these two targets with:

$ vegeta attack -rate=10 -duration=30s -targets=targets.txt | vegeta report

or

$ cat targets.txt | vegeta attack -rate=10 -duration=30s | vegeta report

Generating test traffic

For the purposes of this article we’re going to generate API traffic for a fictional API, the Pokémon collectors club sign up API.

The API resides at https://pokemon-collectors-club.io/sign_up.json accepts a JSON payload via HTTP POST:

{
  "name": "Addicted, New Zealand",
  "email": "login@domain.com",
  "password": "***************",
  "favorite_pokemon_number": 25
}

Testing this API with identical request body data won’t work because the API validates that the email is valid and unique, the password has sufficient complexity, and the “favorite_pokemon_number” is between 1 and 721, so the first request will create a new user, but each subsequent request will return an error.

To generate test traffic we’re going to write a small Go program. Writing a command line program with options (etc) can be tedious, so to help we’ll use a Go package github.com/urfave/cli. And to generate some pretend details we’ll use github.com/icrowley/fake.

Here’s the annotated source code for the program, which I placed in a file named pokemon.go:

package mainimport (  
  "encoding/json"
  "fmt"
  "io/ioutil"
  "math/rand"
  "os"
  "path/filepath"
  "strconv"
  "time""github.com/icrowley/fake"
  "github.com/urfave/cli"
)// Panic if there is an error
func check(err error) {  
  if err != nil {
    panic(err)
  }
}func main() {  
  var (
    users int
  )  // The Go random number generator source is deterministic, so we need to seed
  // it to avoid getting the same output each time
  rand.Seed(time.Now().UTC().UnixNano())  // Configure our command line app
  app := cli.NewApp()
  app.Name = "Pokemon User Data Generator"
  app.Usage = "generate a stream of test data for vegeta. Type 'pokemon help' for details"  // Add -users flag, which defaults to 5
  app.Flags = []cli.Flag{
    cli.IntFlag{
      Name:        "users",
      Value:       5,
      Usage:       "Number of users to simulate",
      Destination: &users,
    },
  }  // Our app's main action
  app.Action = func(c *cli.Context) error {    // Combine verb and URL to a target for Vegeta
    verb := c.Args().Get(0)
    url := c.Args().Get(1)
    target := fmt.Sprintf("%s %s", verb, url)    if len(target) > 1 {      for i := 1; i < users; i++ {        // Generate request data
        name := fake.FullName()
        email := fake.EmailAddress()
        // Password(atLeast, atMost int, allowUpper, allowNumeric, allowSpecial bool)
        password := fake.Password(10, 32, true, true, true)
        pokemonNumber := rand.Intn(720) + 1        // Generate a map of the request body that we'll convert to JSON
        bodyMap := map[string]string{
          "name":                    name,
          "email":                   email,
          "password":                password,
          "favorite_pokemon_number": strconv.Itoa(pokemonNumber),
        }        // Convert the map to JSON
        body, err := json.Marshal(bodyMap)
        check(err)        // Create a tmp directory to write our JSON files
        err = os.MkdirAll("tmp", 0755)
        check(err)        // Use the user's name as the filename
        filename := fmt.Sprintf("tmp/%s.json", name)        // Write the JSON to the file
        err = ioutil.WriteFile(filename, body, 0644)
        check(err)        // Get the absolute path to the file
        filePath, err := filepath.Abs(filename)
        check(err)        // Print the attack target
        fmt.Println(target)        // Print '@' followed by the absolute path to our JSON file, followed by
        // two newlines, which is the delimiter Vegeta uses
        fmt.Printf("@%s\n\n", filePath)
      }
    } else {
      // Return an error if we're missing the required command line arguments
      return cli.NewExitError("You must specify the target in format 'VERB url'", 1)
    }
    return nil
  }  app.Run(os.Args)
}

Before we can run the program, we first need to install its dependencies:

$ go get

Then install it (this will create a binary called pokemon in $GOBIN/).

$ go install

Then run:

$ pokemon -users=10 POST https://pokemon-collectors-club.io/sign_up.json

We should get an output something like this:

POST https://pokemon-collectors-club.io/sign_up.json  
@/Users/me/code/pokemon/tmp/Joshua Henry.jsonPOST https://pokemon-collectors-club.io/sign_up.json  
@/Users/me/code/pokemon/tmp/Mr. Dr. Jimmy Garcia.jsonPOST https://pokemon-collectors-club.io/sign_up.json  
@/Users/me/code/pokemon/tmp/Jeremy Taylor.jsonPOST https://pokemon-collectors-club.io/sign_up.json  
@/Users/me/code/pokemon/tmp/Mr. Dr. Kenneth Bailey.json

Joining the dots

You might have worked out that since this is the same format that vegeta attack requires we can pipe this command directly into the vegeta attack command:

$ pokemon -users=10 POST https://pokemon-collectors-club.io/sign_up.json | \
vegeta attack -rate=10 -duration=30s | vegeta report

This will send our generated data for 10 users to the API at a rate of 10 per second for 30 seconds. We could easily vary the number of users we generate in order to get a higher or lower proportion of new vs returning users.

Next steps

In this article I’ve explained why basic load testing practises don’t always work well for APIs, what Vegeta is and how it works, and how to generate test data and send multiple different request bodies to an API.

I’ve kept the small Go program generic so you should be able to adapt the approaches I’ve taken for your own API load testing.

What other tips and tools do you know for load testing APIs? We’d love to hear — let us know in the comments below!