28 Feb 2015, Bartosz Zmija 28 Feb 2015, Bartosz Zmija Tech World Developer’s tips: Face detection and tracking in app development 24 views Most developers are familiar with popular APIs and development products provided by Google, such as Google Map API or Google Analytics. However, there are lot of services which are far more uncommon, but very helpful in some specific cases and it’s definitely worth to be aware of their existence. In this article I’ve wanted to describe one of them – face detection and tracking API available as part of Mobile Vision APIs. Before diving deep into the subject, it might be a good idea to explain a difference beetwen face detection, tracking and recognition. First term is connected with analysis of static image, in order to find a position of face in it, along with some more specific information, like level of smile of detected object. Tracking is in fact almost the same as detection, although extended for video input. Face recognition, however, is a totally different beast and describes a process of determining whether face matches previously saved pattern (most commonly other face). Unfortunately, this feature is currently unavailable in API provided by Google. To get started, we need to set up Google Play services and add compile ‘com.google.android.gms:play-services-vision:8.3.0’ in our Gradle build file. The basic flow of face detection on given image is very simple. First of all, we need to create a FaceDetector instance using provided FaceDetector.Builder class, which can take different configuration options, such as minimal proportional size of detected face or whether the level of smile should be classified. For example, FaceDetector detector = new FaceDetector.Builder(context) .setClassificationType(FaceDetector.NO_CLASSIFICATIONS) .setLandmarkType(FaceDetector.ALL_LANDMARKS) .setMode(FaceDetector.ACCURATE_MODE) .setProminentFaceOnly(true) .build(); gives us a detector capable of finding specific points of interest on detected face, such as base of the nose or left eye, performing accurate detection and searching only for the largest face on the image. It’s worth noting, that turning on unnecessary features can have serious impact on performance, so we should think which features in our app are… Well, not necessary. Before we use our detector, we need to check whether it’s operational. When Face API is used for the first time, the proper library has to be installed, what might take some time in which detector simply won’t work. At this point we have something that we can use for detecting, but we still need an object to detect from. For this task API provides a special Frame class. To create such object once again we have to use provided builder: Frame frame = new Frame.Builder().setBitmap(bitmap).build(); Then, all we need to do is to get a list of detected Face objects: SparseArray faces = detector.detect(frame); Depending on detector configuration, returned face object will have Landmark list/smiling probability/probability of left and right eye open and the angle of head/any other object on which the face has been detected. After finishing our work with detector, we have to call release() method, as FaceDetector uses native resources which have to be disposed manually. Face tracking is a little bit more complex, as it requires a constant video input. Google has provided some code samples to simplify development, which are available here: https://github.com/googlesamples/android-vision. First of all, we need to declare our FaceTracker, practically in the same way as in previous example. Then, we need to create Processor, which will create Tracker instances capable of providing constant updates of faces on the video. Our processor needs to be assigned to detector in order to work properly with call to the detector.setProcessor() method. However, we still don’t have a source of video. In order to fix that, we need to create appriopriate CameraSource with our detector with code similar to mCameraSource = new CameraSource.Builder(context, detector) .setRequestedPreviewSize(640, 480) .setFacing(CameraSource.CAMERA_FACING_FRONT) .setRequestedFps(30.0f) .build(); This will allow us to track faces from camera. However, we still don’t have any preview. In FaceTracker and multi-processor projects on Github there is CameraSourcePreview class, which renders video from CameraSource on SurfaceView and also gives an opportunity to include own overlay, which will be drawn on top of the rendered view. To properly grasp the whole concept I strongly advise you to take a look on Github projects, because at first all that theory might seem to be a little (over)complicated. Of course, Mobile Vision isn’t only existing solution for face detection and tracking problem. For example, there is OpenCV library, although it’s hard to learn and even harder to master. However, when simple landmark and smile detection isn’t enough, it might be a good idea to give it a try. Why to build your own mobile app? Check out! Bartosz Zmija Android/iOS Developer you might also like 10 Aug 2017, Izabela Majocha Tips & Tricks [QUIZ] How big is your app idea potential? Tips & Tricks 10 Aug 2017, Izabela Majocha [QUIZ] How big is your app idea potential? Do your app idea have potential to achieve a huge success? Complete our quiz and find out! 20 Mar 2017, Supratim Dam Tech World 5 Things We Found Cool At SXSW 2017 Tech World 20 Mar 2017, Supratim Dam 5 Things We Found Cool At SXSW 2017 SXSW is an annual conglomerate of film, interactive media, and music festivals and conferences. Missed it this year? Check our sum-up. 02 Feb 2017, Michał Kowalski Tips & Tricks Why do great mobile app ideas fail? Tips & Tricks 02 Feb 2017, Michał Kowalski Why do great mobile app ideas fail? 77% of apps are abandoned within the first 72 hours after installing. Check how to prevent your app from suffering the same fate.