How to run Android UI tests on virtual devices

viktorbenei · December 15, 2017, 1:30pm

Glad to hear, thanks for reporting @kuassivi!

Indeed; there’s a technical explanation for this, simply that Artifacts can only be uploaded for a specific Build, and when you run the step locally you don’t have any Build URL / “build context” to work with.

If you have any idea for this (where would it upload the artifact from your own local Mac/PC) just let us know!

Right now we’re thinking about allowing the step to run locally (in a “non CI environment”), but in “non CI mode” it would simply print out what files it would upload, it would simply skip the actual upload.

kuassivi · December 16, 2017, 8:34pm

Yup, that was what I would have suggested. There is no real need of moving the files locally.

I also realised you have to create a new step for every different path you want to upload. Why not adding a multiline inputbox like other steps?

Regarding the zip option you have to choose for recursive folders, I can suggest to add a capability like the inline zip extractor functionality of google drive which is based on a js lib that can, perhaps, store temporally the extracted files into the “deployment build directory” and show off the files and paths in a prettify way on the Artifacts tab.

I hope to get some free time to contribute with some steps .

viktorbenei · December 19, 2017, 10:01am

Definitely great ideas, thank you @kuassivi!

I’d suggest you to create a Feature Requests at least for the zip extractor, actually we already have a related one, might worth to just add it as a comment there for better visibility / tracking: Artifacts: Make it possible to view HTML files with relative links (CSS etc. refs) - although zip extractor alone might not solve that issue directly…

If you’d have any questions just let us know!

eric-grab · December 22, 2017, 7:28am

Hi @tamaspapik, it’s working very well, thank you very much!
However, screenshots are missing e.g. https://www.bitrise.io/build/618fca3ce921e377
Are you able to help?
I’m using version 0.9.9 of the step

tamaspapik · December 22, 2017, 8:57am

Hey @eric-grab!

In the build you’ve sent me shall not be any screenshots. Robo test is a test type that makes screenshots automatically, but you’ve ran instrumentation test which will create screenshots only if you code your test to do it. Instrumentation tests also has a video of it, what you can check under the Virtual Device Tests tab

eric-grab · January 10, 2018, 4:21am

Hi @tamaspapik, do you know Spoon and how it can work with this awesome Bitrise step?

I believe the things that need to happen are:

Replace the Gradle task that runs the tests with ./gradlew spoon (I’m using a Gradle plugin; this runs the tests + generate reports)
Copy the reports in “./app/build/spoon-outputs/” to “Apps & Artifacts”

Thanks a bunch

tamaspapik · January 11, 2018, 12:21pm

Hey @eric-grab!

Unfortunately Spoon looks like a client side tool, and to use VDT step, you will need to generate the exact test and app apks.

Just checked Spoon’s documentation, and I can’t see a way of generating them.

eric-grab · January 11, 2018, 12:36pm

Sorry @tamaspapik, let’s forget Spoon for a moment, I’ll rephrase my question as follows:

Replace the Gradle task that runs the tests with ./gradlew someTask
Copy the files in ./app/build/someFolder/ to “Apps & Artifacts” or “Virtual Device Testing” output directory

Is that possible?

tamaspapik · January 11, 2018, 2:17pm

You should be able to replace the gradle task. And also if you have Deploy to Bitrise.io step in your workflow, then files copied to $BITRISE_DEPLOY_DIR should be deployed to the “Apps & Artifacts” tab.

Please contact us on our on-site chat if you encounter any issue doing these, or please send a detailed issue report in the

http://discuss.bitrise.io/c/issues/build-issues

section.

eric-grab · January 12, 2018, 3:38am

Sorry @tamaspapik, how exactly can I “replace the Gradle task”?
There is absolutely no option to configure Gradle for the Virtual Device Testing step

Understood on the use of Deploy to Bitrise.io step, thanks a lot

tamaspapik · January 12, 2018, 12:38pm

So in your VDT step have the two APK path and Test APK path configured. To change gradle task to generate another flavor APKs, you will need to set “tasks” input in Gradle Runner step.

eric-grab · January 13, 2018, 7:37am

How is that possible @tamaspapik? The virtual device/ emulator only lives within the time the Virtual Device Testing step right?
I’ve already tried having Gradle Runner task right after Virtual Device Testing but I got java.lang.RuntimeException: No device(s) found. e.g. https://www.bitrise.io/build/e48b280ad5781812

koral · January 13, 2018, 2:59pm

I believe that @eric-grab asked about Gradle task used to perform the tests like connectedCheck or spoon NOT about one generating APKs (like assemble).

In case of Firebase Test Lab devices are not connected to local ADB server. APKs (testing and tested) need to be uploaded and then you (or step) can execute a command which starts the tests: gcloud firebase test android run | Google Cloud CLI Documentation

eric-grab · January 13, 2018, 3:08pm

Hi @koral, are you saying that behind the scene, Virtual Device Testing step actually calls Firebase Test Lab?

Regardless, I believe I’m very close to achieving my goals, please help me a just a bit more koral and @tamaspapik
Within the life span of Virtual Device Testing step, how can I trigger ./gradlew someTaskIwant?

koral · January 13, 2018, 6:28pm

Exactly:

The tests run on Firebase Test Lab and we display the results we get from them.

Source:

Current version of Virtual Device Testing step waits until tests are finished by polling every 5 seconds:

github.com

bitrise-steplib/steps-virtual-device-testing-for-android/blob/master/main.go#L590


      
          package main
          
          import (
          	"encoding/json"
          	"fmt"
          	"io"
          	"io/ioutil"
          	"net/http"
          	"os"
          	"path/filepath"
          	"strconv"
          	"text/tabwriter"
          	"time"
          
          	"github.com/bitrise-io/go-steputils/stepconf"
          	"github.com/bitrise-io/go-steputils/tools"
          	"github.com/bitrise-io/go-utils/colorstring"
          	"github.com/bitrise-io/go-utils/log"
          	"github.com/bitrise-io/go-utils/pathutil"
          	"github.com/bitrise-io/go-utils/sliceutil"
          	toolresults "google.golang.org/api/toolresults/v1beta3"
          )
          
          const (
          	testTypeInstrumentation = "instrumentation"
          	testTypeRobo            = "robo"
          )
          
          func failf(f string, v ...interface{}) {
          	log.Errorf(f, v...)
          	os.Exit(1)
          }
          
          func main() {
          	var configs ConfigsModel
          	if err := stepconf.Parse(&configs); err != nil {
          		failf("Invalid input: %s", err)
          	}
          
          	if err := configs.validate(); err != nil {
          		log.Errorf("Failed to parse config:")
          		failf("%s", err)
          	}
          
          	fmt.Println()
          	configs.print()
          
          	log.SetEnableDebugLog(configs.VerboseLog)
          
          	fmt.Println()
          	successful := true
          
          	log.Infof("Uploading app and test files")
          
          	testAssets, err := uploadTestAssets(configs)
          	if err != nil {
          		failf("Failed to upload test assets, error: %s", err)
          	}
          	log.Donef("=> Files uploaded")
          
          	fmt.Println()
          	log.Infof("Starting test")
          
          	if err = startTestRun(configs, testAssets); err != nil {
          		failf("Starting test run failed, error: %s", err)
          	}
          	log.Donef("=> Test started")
          
          	fmt.Println()
          	log.Infof("Waiting for test results")
          	{
          		finished := false
          		printedLogs := []string{}
          		for !finished {
          			url := configs.APIBaseURL + "/" + configs.AppSlug + "/" + configs.BuildSlug + "/" + configs.APIToken
          
          			req, err := http.NewRequest("GET", url, nil)
          			if err != nil {
          				failf("Failed to create http request, error: %s", err)
          			}
          
          			client := &http.Client{}
          			resp, err := client.Do(req)
          			if err != nil || resp.StatusCode != http.StatusOK {
          				resp, err = client.Do(req)
          				if err != nil {
          					failf("Failed to get http response, error: %s", err)
          				}
          			}
          
          			body, err := ioutil.ReadAll(resp.Body)
          			if err != nil {
          				failf("Failed to read response body, error: %s", err)
          			}
          
          			if resp.StatusCode != http.StatusOK {
          				failf("Failed to get test status, error: %s", string(body))
          			}
          
          			responseModel := &toolresults.ListStepsResponse{}
          
          			err = json.Unmarshal(body, responseModel)
          			if err != nil {
          				failf("Failed to unmarshal response body, error: %s, body: %s", err, string(body))
          			}
          
          			finished = true
          			testsRunning := 0
          			for _, step := range responseModel.Steps {
          				if step.State != "complete" {
          					finished = false
          					testsRunning++
          				}
          			}
          
          			msg := ""
          			if len(responseModel.Steps) == 0 {
          				finished = false
          				msg = fmt.Sprintf("- Validating")
          			} else {
          				msg = fmt.Sprintf("- (%d/%d) running", testsRunning, len(responseModel.Steps))
          			}
          
          			if !sliceutil.IsStringInSlice(msg, printedLogs) {
          				log.Printf(msg)
          				printedLogs = append(printedLogs, msg)
          			}
          
          			if finished {
          				log.Donef("=> Test finished")
          				fmt.Println()
          
          				log.Infof("Test results:")
          				w := tabwriter.NewWriter(os.Stdout, 0, 0, 3, ' ', 0)
          				if _, err := fmt.Fprintln(w, "Model\tAPI Level\tLocale\tOrientation\tOutcome\t"); err != nil {
          					failf("Failed to write in tabwriter, error: %s", err)
          				}
          
          				for _, step := range responseModel.Steps {
          					dimensions := map[string]string{}
          					for _, dimension := range step.DimensionValue {
          						dimensions[dimension.Key] = dimension.Value
          					}
          
          					outcome := step.Outcome.Summary
          
          					switch outcome {
          					case "success":
          						outcome = colorstring.Green(outcome)
          					case "failure":
          						successful = false
          						if step.Outcome.FailureDetail != nil {
          							if step.Outcome.FailureDetail.Crashed {
          								outcome += "(Crashed)"
          							}
          							if step.Outcome.FailureDetail.NotInstalled {
          								outcome += "(NotInstalled)"
          							}
          							if step.Outcome.FailureDetail.OtherNativeCrash {
          								outcome += "(OtherNativeCrash)"
          							}
          							if step.Outcome.FailureDetail.TimedOut {
          								outcome += "(TimedOut)"
          							}
          							if step.Outcome.FailureDetail.UnableToCrawl {
          								outcome += "(UnableToCrawl)"
          							}
          						}
          						outcome = colorstring.Red(outcome)
          					case "inconclusive":
          						successful = false
          						if step.Outcome.InconclusiveDetail != nil {
          							if step.Outcome.InconclusiveDetail.AbortedByUser {
          								outcome += "(AbortedByUser)"
          							}
          							if step.Outcome.InconclusiveDetail.InfrastructureFailure {
          								outcome += "(InfrastructureFailure)"
          							}
          						}
          						outcome = colorstring.Yellow(outcome)
          					case "skipped":
          						successful = false
          						if step.Outcome.SkippedDetail != nil {
          							if step.Outcome.SkippedDetail.IncompatibleAppVersion {
          								outcome += "(IncompatibleAppVersion)"
          							}
          							if step.Outcome.SkippedDetail.IncompatibleArchitecture {
          								outcome += "(IncompatibleArchitecture)"
          							}
          							if step.Outcome.SkippedDetail.IncompatibleDevice {
          								outcome += "(IncompatibleDevice)"
          							}
          						}
          						outcome = colorstring.Blue(outcome)
          					}
          
          					if _, err := fmt.Fprintln(w, fmt.Sprintf("%s\t%s\t%s\t%s\t%s\t", dimensions["Model"], dimensions["Version"], dimensions["Locale"], dimensions["Orientation"], outcome)); err != nil {
          						failf("Failed to write in tabwriter, error: %s", err)
          					}
          				}
          				if err := w.Flush(); err != nil {
          					log.Errorf("Failed to flush writer, error: %s", err)
          				}
          			}
          			if !finished {
          				time.Sleep(5 * time.Second)
          			}
          		}
          	}
          
          	if configs.DownloadTestResults {
          		fmt.Println()
          		log.Infof("Downloading test assets")
          		{
          			url := configs.APIBaseURL + "/assets/" + configs.AppSlug + "/" + configs.BuildSlug + "/" + configs.APIToken
          
          			req, err := http.NewRequest("GET", url, nil)
          			if err != nil {
          				failf("Failed to create http request, error: %s", err)
          			}
          
          			client := &http.Client{}
          			resp, err := client.Do(req)
          			if err != nil {
          				failf("Failed to get http response, error: %s", err)
          			}
          
          			if resp.StatusCode != http.StatusOK {
          				failf("Failed to get http response, status code: %d", resp.StatusCode)
          			}
          
          			body, err := ioutil.ReadAll(resp.Body)
          			if err != nil {
          				failf("Failed to read response body, error: %s", err)
          			}
          
          			responseModel := map[string]string{}
          
          			err = json.Unmarshal(body, &responseModel)
          			if err != nil {
          				failf("Failed to unmarshal response body, error: %s", err)
          			}
          
          			tempDir, err := pathutil.NormalizedOSTempDirPath("vdtesting_test_assets")
          			if err != nil {
          				failf("Failed to create temp dir, error: %s", err)
          			}
          
          			for fileName, fileURL := range responseModel {
          				err := downloadFile(fileURL, filepath.Join(tempDir, fileName))
          				if err != nil {
          					failf("Failed to download file, error: %s", err)
          				}
          			}
          
          			log.Donef("=> Assets downloaded")
          			if err := tools.ExportEnvironmentWithEnvman("VDTESTING_DOWNLOADED_FILES_DIR", tempDir); err != nil {
          				log.Warnf("Failed to export environment (VDTESTING_DOWNLOADED_FILES_DIR), error: %s", err)
          			} else {
          				log.Printf("The downloaded test assets path (%s) is exported to the VDTESTING_DOWNLOADED_FILES_DIR environment variable.", tempDir)
          			}
          		}
          	}
          
          	if !successful {
          		os.Exit(1)
          	}
          }
          
          func downloadFile(url string, localPath string) error {
          	out, err := os.Create(localPath)
          	if err != nil {
          		return fmt.Errorf("Failed to open the local cache file for write: %s", err)
          	}
          	defer func() {
          		if err := out.Close(); err != nil {
          			log.Printf("Failed to close Archive download file (%s): %s", localPath, err)
          		}
          	}()
          
          	resp, err := http.Get(url)
          	if err != nil {
          		return fmt.Errorf("Failed to create cache download request: %s", err)
          	}
          	defer func() {
          		if err := resp.Body.Close(); err != nil {
          			log.Printf("Failed to close Archive download response body: %s", err)
          		}
          	}()
          
          	if resp.StatusCode != 200 {
          		return fmt.Errorf("Failed to download archive - non success response code: %d", resp.StatusCode)
          	}
          
          	_, err = io.Copy(out, resp.Body)
          	if err != nil {
          		return fmt.Errorf("Failed to save cache content into file: %s", err)
          	}
          
          	return nil
          }
          
          func uploadFile(uploadURL string, archiveFilePath string) error {
          	archFile, err := os.Open(archiveFilePath)
          	if err != nil {
          		return fmt.Errorf("Failed to open archive file for upload (%s): %s", archiveFilePath, err)
          	}
          	isFileCloseRequired := true
          	defer func() {
          		if !isFileCloseRequired {
          			return
          		}
          		if err := archFile.Close(); err != nil {
          			log.Printf(" (!) Failed to close archive file (%s): %s", archiveFilePath, err)
          		}
          	}()
          
          	fileInfo, err := archFile.Stat()
          	if err != nil {
          		return fmt.Errorf("Failed to get File Stats of the Archive file (%s): %s", archiveFilePath, err)
          	}
          	fileSize := fileInfo.Size()
          
          	req, err := http.NewRequest("PUT", uploadURL, archFile)
          	if err != nil {
          		return fmt.Errorf("Failed to create upload request: %s", err)
          	}
          
          	req.Header.Add("Content-Length", strconv.FormatInt(fileSize, 10))
          	req.ContentLength = fileSize
          
          	resp, err := http.DefaultClient.Do(req)
          	if err != nil {
          		return fmt.Errorf("Failed to upload: %s", err)
          	}
          	isFileCloseRequired = false
          	defer func() {
          		if err := resp.Body.Close(); err != nil {
          			log.Printf(" [!] Failed to close response body: %s", err)
          		}
          	}()
          
          	_, err = ioutil.ReadAll(resp.Body)
          	if err != nil {
          		return fmt.Errorf("Failed to read response: %s", err)
          	}
          
          	if resp.StatusCode != 200 {
          		return fmt.Errorf("Failed to upload file, response code was: %d", resp.StatusCode)
          	}
          
          	return nil
          }

This file has been truncated. show original

It should be possible to split this step into separate “Upload APKs and start tests on Virtual devices” and “Retrieve Virtual devices test results”. Between them you can insert whatever step you want.

eric-grab · January 14, 2018, 8:04am

Thanks @koral

Reading main.go and Firebase Test Lab documentation, it’s unclear to me whether you guys are calling a REST API (not found in their documentation) or Gcloud CLI (doesn’t seem to support custom Gradle task).
Can’t see myself being able to have both Virtual Device Testing + Spoon working together any time soon.
Will put this off for a while.
Thanks a lot for your help thus far @koral and @tamaspapik

tamaspapik · January 19, 2018, 2:15pm

Firebase REST API doesn’t support stuff like that, we have our own server to handle requests, so this is why you won’t see our calls in the official documentation Thanks @koral!

So basically VDT uses an “offline” solution to test, which means that it prepares/uploads all stuff required for the test, and then runs the given configuration. not like you have a device connected that can be used from adb. I think this is why you won’t be able to use VDT for this case

bootstraponline · January 22, 2018, 11:14pm

Checkout https://github.com/TestArmada/flank for a custom FTL runner that works on bitrise. Spoon is not FTL compatible.

sah · January 25, 2018, 10:11pm

This is great. Been struggling with the Create Emulator, Start Emulator, etc. until I noticed this step.

Is it supported to generate jacoco report from the instrumental tests?

tamaspapik · January 26, 2018, 11:43am

Glad to hear that!

Does your jacoco report is generated to the device’s sdcard or so? In that case you can fill the pull directories input of the step, so it will be exported to the test artifacts.

Topic		Replies	Views
Android UI Test Tooling Issues	2	692	June 3, 2021
Is it possible to use mapping.txt from Gradle Runner in combination with virtual UI device testing? Question & Answer android	2	588	September 29, 2021
Android Virtual Device Testing: Not running UI tests for all modules Builds testing	5	2623	March 14, 2018
Android: Virtual Device Testing on multiple modules Steps	9	3829	September 27, 2021
Android Build for UI Testing v0.1.0 Changelog	1	843	April 11, 2019

How to run Android UI tests on virtual devices

Related topics