Features
Start a free trial

Mark test status with Playwright

The Playwright tests you run on TestingBot are using the WebSocket protocol to send commands to a remote browser. These commands instruct the remote browser what to do: click, type, resize, ...

Because the logic of your test cases is inside your test script, TestingBot has no way of knowing if a test failed or passed.
There is a way to send back the test status to TestingBot, by including a code snippet at the end of your test, where you know whether the test passed or failed.

Below we'll go over two methods to send back the test status to TestingBot, so that we can show the pass/failure state in the dashboard and in the TestingBot analytics.

Mark test status from within a test script

TestingBot has created a custom Playwright command which you can use in your Playwright scripts. You can pass in a boolean to indicate whether the test passed or failed. 

await page.evaluate(_ => {}, `testingbot_executor: ${JSON.stringify({action: 'setSessionStatus',arguments: { passed: false, reason: 'Title did not match' }})}`)

This command will set the test as failed, with a specific reason which will be visible on the TestingBot test detail page.

You can pass in the reason for a failure with the reason field.

This command will return a JSON object with the sessionId of the test, which can be used with our REST-API.

You would typically call this at the end of each test, where you know if the test failed or passed. For example, see the code snippet below where we use chai's expect to verify if the test passed or not. 

const pw = require('playwright-core')
const expect = require('chai').expect

(async () => {
    const browser = await pw.chromium.connect({
        wsEndpoint: 'wss://cloud.testingbot.com/playwright?key=api_key&secret=api_secret&browserName=chrome&browserVersion=latest',
    })
    const context = await browser.newContext()
    const page = await context.newPage()

    await page.goto('https://testingbot.com/')

    try {
        expect(title).to.equal("Cross Browser Testing and Mobile App Testing. Test on web browsers, mobile emulators, simulators and physical mobile devices.", 'Expected page title is incorrect!')
        // This is ok, so mark the test as passed
        await page.evaluate(_ => {}, `testingbot_executor: ${JSON.stringify({action: 'setSessionStatus',arguments: {passed: true, reason: 'Title matched'}})}`)
      } catch {
        // Test failed, mark the test as failed on TestingBot
        await page.evaluate(_ => {}, `testingbot_executor: ${JSON.stringify({action: 'setSessionStatus',arguments: {passed: false, reason: 'Title did not match'}})}`)
      }
    await browser.close()
})()

Mark test status with API

You can fetch the sessionId during an active Playwright test. This unique id can be used during or after the test to set the pass/fail state for the test.

Marking the test as passed or failed will show the success state in the TestingBot test dashboard, on the individual test detail page and in the TestingBot analytics pages.

To fetch the sessionId, please use the sample code below. 

const testingBotResponse = await page.evaluate(_ => {}, `testingbot_executor: ${JSON.stringify({action: 'getSessionDetails'})}`)

The testingBotResponse variable will contain a sessionId property which you can use: 

{
    "sessionId": "......"
}

With the sessionId, you can now mark the test as passed or failed via the TestingBot REST-API: 

$ curl "https://api.testingbot.com/v1/tests/:sessionId" \
-X PUT \
-d "test[success]=1" \
-u key:secret

Learn more about this API call and other API calls on the TestingBot API documentation.