Automated Testing (Android Game Developer Summit 2018)
Articles Blog

Automated Testing (Android Game Developer Summit 2018)

September 16, 2019

XIN LIU: Hi, everyone. I’m Xin from NetEase games. Today I’m going to talk
about Airtest project, the next generation automated
test team for games. Two month ago, on
JDC we launched here an open source-based project. So how many of you
here have heard of it? Oh. Pretty much. OK. Today I’m going to talk
about these three topics– challenges in testing,
and how Airtest project can help you solve
these problems; and also our internal priorities of
larger scale real device testing. So the first issue is
that we have so many games to test in NetEase games. These are games we developed
and released since 2014. We have released over
150 mobile games. And also, we have run several
PC games for over 10 years. So how do we ensure the high
quality of all these games? Another issue is Android
device fragmentation. And because Android
system is open source, and OEMs can customize
their own Android phones. And this graph shows
the situation in China. You can see Samsung,
Huawei, Xiaomi. Each have dozens
of Android models, and each model has
different screen resolution, different system APIs,
or even graphic drivers. in NetEase games,
we usually need to test more than 200 Android
phones before releasing a game. Then let’s take a look at
the difference of testing between apps and the games. The first one is
usually our games are released on
multiple platforms, like Android, iOS; the
same version of game on different platforms. And sometimes on desktop. And just now we said
Chrome OS, right. And sometimes Tim HTML5
platforms, console, or even VR. The next difference is
that we got less support from a third party. I mean, such as test
frameworks, or testing tools, and platforms. Unlike apps, Google
provide app developers with test frameworks, like UI
Automator, or is [INAUDIBLE].. And also testing tools are
integrated in Android Studio. Now also we have Firebase
Test Lab for app testing. Another big issue is that games
always need more test cases, because users, there
are more content that can be played in
a game than in a app. So the test cases are increases
exponentially in games. So how do we solve these issues? We can hire more and more
people to do testing. But hiring more
people to do test on 200 Android phones, that’s
not tolerable, I think– even if it’s in China. So automation is here to help. Actually, we have developed
this project for three years internally. And last year on Google I/O, I
talked with Firebase Test Lab team. And we collaborate open
sourcing these project. So let me introduce
Airtest IDE first. It’s a desktop IDE. The right side is
the mirror window of an Android phone connecting
with your PC via ADB. And the middle part
is a code editor, where you can write
arbitrary Python code. And the left part
shows the hierarchy of this, the UI hierarchy
of this Unity game, and also the APIs provided
by our test framework. And when you operate
on the screen window, test code will be automatically
generated in the Code Editor. Let’s see the demo. Yeah. Click the recording button. And perform a touch
action, it will generate a sentence in Python. Also, you can edit it. Swipe action. And then we can slip one second. Also, make assertions
of the UI show up. Then we can run it
immediately on this phone to see if it works. After you run the tests– oh, yeah, it worked– after you run it, you can
check the HTML report. It shows every
step of your tests. If any step went wrong,
it will be labeled red, and you can see
the failure point. You may have noticed that
we have two underlying test framework here. The first is Airtest framework. It uses image recognition
[INAUDIBLE] knowledge to locate UI elements. And then in the use device APIs
to perform simulated input. This is the structure
of a Airtest. We provide users with a simple
test API, like simulated input, like make a insertion. And then it uses image
recognition to locate a UI. And then we have a underlying
abstract layer of platform API. We unify the simulate the input
APIs of different platforms so that user can
run their script on different platforms, like
Android, iOS, and Windows, and VR. There is another
framework called Poco. This framework is similar to
the task frameworks for apps, like UI Automator for Android. But most of the games
uses graphic API, like OpenGL or Vulkan
to render the UI widget. So we cannot directly use UI
automator on Android for games. So we developed Poco. This is how it works. We also have a
underlying abstract layer of device abstraction here
for different game engines. We provide each game
engine an SDK so that the test framework
can communicate with the game engine. We are JSON RPC. And then we unify
different game engines. So we can write test for
different game engines. Also, the custom engines. In the morning I saw most of
you are using custom engines. We provide multiple
language SDK so you can implement in
your own game engine, and start using our test
framework, and also our tools. Here is a comparison. Airtest use image
recognition to locate UI, and use platform API to
perform simulated input. And what’s cool
here is it does not require any instrumentation. Just plug in your
phone, and start using. And we support
multiple platforms. And Poco uses UI
hierarchy inspection, and use JSON RPC to communicate
with the SDK in game engine. Now you need to
integrate our SDK. It usually takes
less than 10 minutes. And now we support Unity and
Cocos, those two game engine. And also Android native
app, we implement an SDK using accessibility service. So you don’t need to
integrate anything into app for Android native
apps, and also custom game engine. So how do we choose
which framework to use? Here are some
suggestions from us. Actually, in NetEase,
we have written thousands of tests for different
type of games internally. And the advantage of
Airtest is obvious. It requires no instrumentation. And using open CV
to do assertion, to make assertion of
the UI, is reliable, because it checks exactly what
users are expected to see. And for Poco, since it uses
UI hierarchy inspection, it can be more accurate,
especially for cases of 3D object when the object change
their orientation in games. And this Poco is similar
to test frameworks for apps like UI Automation. And it covers the what
is missing for games. And we released these two
project GDC two month ago. And we got 6,000 downloads,
and over 1,000 stars in this two month. And please try our project,
and star us on GitHub. But it’s started. We have two more new
features in this two month. First we support iOS. Actually, we have
supported iOS last year, but the performance is not
good enough to be released. Let’s see this demo. Connecting your iOS
with your x code. Then you can operate the screen
just like for Android phones. And this is one of our games. Then you can run the
tests on this iPhone. That’s it. The back panel of
this iOS support is originated from
Facebook web travel agent. But the web travel agent, the
problem is that it’s too slow. Simple touch action can
take as long as one second, and it’s not tolerable
for games automation. So we optimize most
of its API last month. And now the frame rate can be
up to 15 FPS in the Airtest IDE. Another feature is we
support web pages automation. You can open the Chrome
browser in the IDE. And then you can
operate on the Chrome. And then code will be
automatically generated. Yeah. The [INAUDIBLE] is based
on Selenium framework. Also you can use
Airtest API to make insertions to run the test,
and then generate the report. Since Selenium is quite immature
in web pages automation, we have no reason
to reinvent it. So we just wrote a
plugin for Airtest IDE to help recording this
web pages automation. I think it will be useful
if you are releasing game on HTML platform. Until now we talked
about Airtest project. You can use Airtest IDE
to record and run the test on your PC and on your phone. But how do we test
hundreds of Android phones? The first thing is
Firebase Test Lab. Firebase Test Lab is a
test service provided by Google Firebase team. And what do we do
collaborating with their team is we support running
Airtest and the Poco script on Firebase Test Lab. And you can use Airtest
IDE to bundle your test script into an APK. And then you can upload
it to the FTL web page. And then you can start
a instrumentation test, and use the cloud of
devices provided by FTL. And this is our
internal device farm. We have set up a device
farm of 200 devices– 200 Android devices. And we have written now the
number could be 2,000 scripts. And we run those scripts of
different games every week, because our game
are updated weekly. And let’s see how it works. It’s just [INAUDIBLE]
and around the test generated by Airtest IDE. Yeah, that’s it. As for our roadmap for
this automation project, we intended to
support more clients– iOS, Android Emulator,
and web pages. These are in better version. You can try it now. And feel free to fire us a
issue if you find any bug. And also we are Hybrid apps, and
UnrealEngine is in the future. Also, we supported Android TV. But I’m not sure
about Chrome OS. But maybe it works. And also another
important thing is we want to build
up the open source community for this project, and
I hope more and more developers like it, and get
involved in this project. Yeah, thank you. And any question? AUDIENCE: You
mentioned you’re using the accessibility service. Did you have any
issue with that when that was restricted recently? XIN LIU: When it what? AUDIENCE: The
accessibility service. It was restricted for apps that
were using those launchers, or for any apps that were
trying to access the reverse [INAUDIBLE] tapped in large. Those are things that were
being restricted because they were being taken advantage of. Have you guys seen
any impact from that? XIN LIU: For UI
testing of native apps, we don’t see any restriction. I think it can do what UI
automation or [INAUDIBLE] do.

Only registered users can comment.

  1. Is there a way to run automation on more than one window at a time?
    I am aware that focus can only be given to one window, however I suppose I am curious about possibly dockerizing this, or a light-weight way of running multiple instances?
    Regardless this is an amazing product you have created!
    Been looking for something like this for awhile… I've used lackey before, but this is far superior. (Airtest)

Leave a Reply

Your email address will not be published. Required fields are marked *