Facial Recognition The new science of identity

More than our signature, fingerprint or even voice, our face is the most fundamental form of identification. With technology now able to capture, analyze and compare data about the distance between forehead and chin, or the contour of our eye sockets, we take the application of facial recognition very personally. And rightly so. In the online economy, where activity happens remotely, reliable authentication and identification can be hard to achieve. An email is not a failsafe proof of identity, people forget passwords and PINs, which is why attention has turned to biometric alternatives and the potential of facial recognition.

The difference between biometric authentication and identification

Biometric authentication stores a person’s characteristics in a database or in a secure portable element such as a smart card or increasingly in a secure chip inside your mobile phone. This digital file could be a photo of a face, a voice recording or fingerprint image. It is then compared to the person’s live biometric data. If there’s a match, the person’s identity is verified.

Biometric identification determines the identity of an individual. Here, the live biometric data is compared with the data of many (possibly thousands of) other people. The system then confirms the correct match.

Of all the biometric options, facial recognition has attracted the most excitement – especially because of recent advances in the technology.

Early systems struggled in low-light conditions. Spoofers could trick them with photographs and videos. Now, that has changed. Today’s systems use technology such as 3D mapping to achieve live biometric authentication and deliver compelling levels of accuracy.

Because of this, smartphone and tablet makers now use facial identification as the default ‘unlock’ method for their devices and services. Indeed, Counterpoint Research expects more than a billion smartphones with facial recognition will ship in 2020.

Facial recognition has applications beyond authentication. Organizations can use it for surveillance or to analyze consumers’ emotions in store.

Facial recognition has traditionally been associated with the security sector. However, facial recognition is now expanding across many other industries. For example, it is an emerging technology for the future of mass transportation, with potential uses for safety in urban environments, according to a report by Visa and Stanford School of Engineering. All of which is fueling a market that will be worth $5.38 billion by 2020, according to Technavio.

In this report, we will look more deeply into the technology, its many real-world applications and why the ethics of facial recognition need to evolve in tandem with its applications.

DOWNLOAD THE FACIAL RECOGNITION WHITEPAPER HERE

Back to top

Why secure and reliable authentication matters... And how facial recognition can provide it

In a world of digital services, issues of identity and authentication pose a problem. How can people trust each other when they are not physically present?

Clearly, traditional paper-based forms of identity – passport, driver’s license, ID card – are not the answer. These documents were designed to be checked in physical settings by human agents on bespoke machines. And although they have been updated with contactless technology, checking paper documents can be time-consuming.

Meanwhile, many types of authentication are flawed, especially passwords. With the average person needing to keep track of multiple passwords, many of which may need updating regularly, the password has become impractical as well as insecure.

Criminals can ‘phish’ people out of their passwords. Or they can use ‘brute force’ software to try millions of word combinations every second. The problem is that a widespread, universal (and convenient) form of digital identity doesn’t currently exist.

There’s also a sense that facial recognition is best suited for future work and leisure patterns.

“The face is the most flexible biometric authentication modality,” says Alan Goode, CEO and Chief Analyst of Goode Intelligence. “It can be used in different contexts and settings. There’s no need for sensors. And this fits with current megatrends. Take ride sharing. If we move into a future of autonomous driving and shared ownership, we will need new and seamless ways to identify ourselves. Facial recognition seems to provide the best answer to this.”

Facial recognition is democratic. According to the World Bank, there are more than 1.1 billion individuals without official proof of identity. This is a major challenge. Without an officially sanctioned identity, it can be difficult for people to access finance, healthcare, social protection and even work.

But on a wider level, it’s just more convenient to have a single digital identity. Frederic Trojani, chairman of the Security Identity Alliance, says: “Today, it seems we need a new identity every time we sign up for a new service. What we need is a sovereign digital identity – given to us at birth by our government – which is interoperable across all consumer, enterprise and government domains. This will provide a secure way to authenticate us with any entity that needs to know who we are, before offering benefits or services.”

The difference between identity and authentication

It’s tempting to use the words identity and authentication interchangeably. This is wrong and it’s important to understand the differences.

Identity: A set of credentials that comprises ‘you’. This could be your name and address. This could be a token or alias (an email address or phone number). Sometimes, this is even a physical ‘thing’ (an ATM card). Your identity must be unique and say to a third party ‘this is me’.

Authentication: What we do when a third party asks: ‘How can you prove it?’. In the case of an ATM card, the card is the ID and authenticator is the PIN. In the case of border control, the person’s ID is their name and passport. The authentication is carried out by the agent who looks at a person’s face and certifies the likeness.

A person’s identity is unique, but it is not secret. Authentication keys must be either secret (PINs, passwords) or hard to copy.

With biometric authentication, the authenticator is not secret, but it is difficult to mimic. In the case of facial recognition, a system must be able to distinguish between a genuine subject and a faker using a photo, mask or video clip. So the ability of the authentication method to deter these attacks determines how strong it is.

DOWNLOAD THE FACIAL RECOGNITION WHITEPAPER HERE

Back to top

Facial recognition: how the technology has grown up

Facial recognition is more than 50 years old. A research team led by Woodrow W Bledsoe ran experiments between 1964 and 1966 to see whether ‘programming computers’ could recognize human faces. The team used a rudimentary scanner to map the locations of the person’s hairline, eyes and nose. The task of the computer was to find matches.

It wasn’t successful. Bledsoe said: “The face recognition problem is made difficult by the great variability in head rotation and tilt, lighting intensity and angle, facial expression, ageing, etc.”

In fact, computers find it harder to recognize faces than beat Grandmasters at chess. It would be many years before these problems were overcome.

Thanks to improvements in camera technology, mapping processes, machine learning and processing speeds, facial recognition has come of age.

Most systems use 2D camera technology, which creates a flat image of a face, and maps ‘nodal points’ (size/shape of eyes, nose, cheekbones etc.). The system then calculates the nodes’ relative position and converts the data into a numerical code. The recognition algorithms search a stored database of faces for a match.

2D technology works well in stable, well-lit conditions such as passport control. But it is less effective in darker spaces and cannot deliver good results when the subjects move around. It is easy to spoof with a photograph.

One way to overcome these flaws is with liveness detection. These systems will look for indicators of a non-live image such as inconsistent features between foreground and background. They may ask the user to blink or move. They are needed to defeat criminals who try to cheat facial recognition systems by using photographs or masks.

Another key advance is the ‘deep convolutional neural network’. This is a type of machine learning in which a model finds patterns in image data. It deploys a network of artificial neurons that imitates the functioning of the human brain. In effect, the network behaves like a black box. It is given input values whose results are not yet known. It then makes checks to ensure the network is producing the expected result. When this is not the case, the system makes adjustments until it is correctly configured and can systematically produce the expected outputs.

Today, previously advanced processes are finding their way into mass market devices. For example, Apple uses 3D camera tech to power the thermal infrared-based Face ID feature in its iPhone X. Thermal IR imagery maps the patterns of faces derived primarily from the pattern of superficial blood vessels under the skin.

Apple also sends the captured face pattern to a ‘secure enclave’ in the device. This ensures the authentication happens locally and that the patterns are not accessible by Apple.

Measurements and accuracy

Facial recognition systems are assessed by three criteria.

1. False positive (aka false acceptance)

This describes when a system erroneously makes an incorrect match. The number should be as low as possible.

2. False negative (aka false rejection)

With a false negative, a genuine user is not matched to his or her profile. This number should also be low.

3. True positive

This describes when an enrolled user is correctly matched to his or her profile. This number should be high.

These three measurements are conveyed in percentages. So, let’s say an entry system assesses 1,000 people a day. If five non-approved people are allowed in, the false positive rate is five in 1,000. That’s one in 200 or 0.5%.

So, what percentages do the current systems achieve? The National Institute of Standards and Technology (NIST) regularly tests multiple systems to search a database of 26.6 million photos.

In its 2018 test, it found that just 0.2% of searches failed to match the correct image, compared with a 4% failure rate in 2014. That’s a 20x improvement over four years.

NIST computer scientist Patrick Grother says: “The accuracy gains stem from the integration, or complete replacement, of prior approaches with those based on deep convolutional neural networks. As such, face recognition has undergone an industrial revolution.”

Further confirmation of the improvement in the tech came in the Department of Homeland Security’s Biometric Technology Rally in 2018. In its test, Gemalto’s Live Face Identification System (LFIS) scored a 99.44% acquisition rate in under five seconds, compared to the average of 65%.

Facial recognition vs face detection: an important difference

Though ‘facial recognition’ is generally used as a catch-all term, this is not entirely accurate. There is a key distinction between facial recognition and face detection.

Facial recognition describes the process of scanning a face and then matching it to the same person on a database. This is the approach used to unlock phones or authenticate a person entering a building.

Face detection is when a system tries simply to establish that a face is present. Social media companies use face detection to filter and organize images in large catalogues of photos, for example.

The tools used to train the two systems are different. The desired levels of accuracy vary too. Clearly, facial recognition used for identification purposes needs to score more highly than any system used to merely organize images.

The confusion between the two processes has caused some controversy. In 2019, a researcher revealed that Amazon’s systems were much better at classifying the gender of light-skinned men than of dark-skinned women. This led to fears that surveillance systems could make more false matches for some ethnic groups. However, Amazon countered that the error rates related to face detection, which is not used to identify specific individuals.

Where is facial recognition in action?

Reaching new heights in airport boarding

Every year, more than 100 million passengers pass through Paris-Charles de Gaulle and Paris-Orly airports. In 2009, the airports’ owner – the ADP Group – introduced PARAFE fingerprint recognition technology to speed up the process. However, only 4% of people could use it, as the access to fingerprint biometrics were protected within the ePassports’ chip. And there was often confusion among users. Which finger? Which hand?

Of course, the new systems complement ‘human’ security measures. Border police can still monitor passengers in real time, make random checks and block the opening of gates.

Since 2018, many more airlines have started to use on-board facial recognition technology. In the US alone, more than 15 airports have set up face matching systems to help board passengers faster and more safely. Trials have taken place at Los Angeles and San Jose airports, among many others.

The technology is creating a lot of interest in Europe too. For example, the airport specialist Amadeus has successfully tested facial recognition technology for flights operated by Adria Airways, Air France and LOT Polish Airlines at Ljubljana airport.

It found that the introduction of biometrics has also cut the average boarding time. The trial was conducted across 15 flights on which 175 passengers volunteered to enroll with their passport or ID card and a selfie-check. The live picture was sent to a server and then used to match the passenger when he or she passed in front of the camera at the departure gate. It took fewer than two seconds. With this simple enrolment of their data by smartphone, 98% of passengers were matched successfully and able to board their flight.

Biometric border checks in Europe

The Schengen Area comprises 26 states in mainland Europe. Inside the zone, there are no passport or border controls. This is convenient for nationals that live inside it. But it presents a problem for border guards who will need to track the forecasted 887 million non-EU citizens visiting the area every year by 2025.

At present, there is no central system for recording all entries to, or exits, from the EU-Schengen zone. The guards of each member state stamp passports with dates of entry and exit, but they do not record cross-border movements. The stamps they use can be counterfeited or difficult to read. There is no electronic record kept, and no automatic calculation of the duration of the authorized stay. Instead the guards make these calculations manually.

Now, an upgrade is underway. The Large IT Systems Agency (EU-LISA) is developing an entry/exit system to record the biometric data of all non-EU nationals crossing the external borders of the EU. This will include facial data.

The new system should abolish the need for manual stamping of passports. It should help guards to detect travelers attempting to use multiple identities. Other bodies such as Europol will have regulated access to the data too. The system is set to be operational in 2020.

Unlocking your cash in Australia Bank

Bank customers are familiar with using facial identification to unlock their mobile apps. But ATMs? That’s still the domain of plastic cards and PINs. Regrettably, there is a lot of fraud in this area. Criminals routinely use skimming devices to hack into ATMs.

In 2018, National Australia Bank and Microsoft joined forces to test a system for unlocking cash machines with a face scan. The partners did not store images locally and erased all biometric data following the experiment.

Steven Worrall, Managing Director of Microsoft Australia, said: “We want to deliver great connected customer experiences. It’s a look into what the future might hold for the way our customers access banking products and services.”

Home run for Major League Baseball

Getting thousands of people in and out of sports stadia safely and quickly is a huge challenge. Digital ticketing and payment systems have helped. The next stage is a fully biometric entry/exit process. In the US, Major League Baseball (MLB) is already working on this. The MLB is partnering with tech specialist CLEAR to introduce biometric authentication scanning at participating ballparks in 2019. Visitors who link their CLEAR profile with their MLB.com account can gain entry with a finger press and, in the near future, facial recognition technology. Fans can also use the system to pay for refreshments and validate their age.

The Mastercard selfie

Existing identity verification methods for e-commerce present problems for both shoppers and merchants. Shoppers forget their passwords or abandon purchases when the sign-in process is too time-consuming. Merchants lose business whenever this happens. To address this, Mastercard turned to facial recognition. In 2016, it introduced Identity Check Mobile. Here, the cardholder takes a selfie on their smartphone to securely – and quickly – authenticate their identity with the card network and the retailer.

Protecting problem gamblers

Many casinos now give problem gamblers the option to self-exclude from their locations. Now, they are using facial recognition to deny entry to these individuals. The system monitors people entering casinos and checks their face against a voluntary photo database of problem gamblers who have requested to be barred from specific venues. Staff are alerted to check the person’s ID when the system finds a match.

Australia’s Crown Casino is one of these and says the system worked even when a self-excluded gambler tried to disguise his appearance.

Speeding up hotel check-in

For all the tech in hotel rooms and booking systems, check-in is still stuck in the 1960s. There are queues. It can take ages. With biometrics to play a significant role in the hospitality industry, enabling guests to skip front desk lines and cutting the check-in process, hotel chains are prepared to jump in, just like Marriott, which recently started testing facial recognition.

Buying burgers

In recent years, fast food outlets have been busy replacing their human cashiers with self-service kiosks. The next logical step could be letting people order with a simple face scan. In California, the burger chain CaliBurger is already testing such a set-up. Customers at its Pasadena outlet can use its kiosks to order, pay and log into the chain’s loyalty/rewards program. “Our goal is to replace credit card swipes with face-based payments,” said the firm’s CEO John Miller. Biometric payments are already flourishing in China. Motivated by positive customer feedback, hundreds of restaurants across the country have rolled out digital technologies to improve customer experiences, growth, and operational efficiency.

Making an entrance at events

The 2019 Brit Awards used facial recognition technology to enhance its event security, deploying it to screen guests at multiple entrances to the O2 in London. The system linked to mobile apps, which enabled trained staff members to make secondary face-to-face identity verification checks.

Driving the future

After 100 years of relatively incremental change to cars, the sector is full of innovation. Facial recognition is of particular interest to manufacturers. China’s Byton, one of the newer car makers, has revealed a concept model that lets drivers unlock it with a simple face scan.

Looking to the future, as the automotive industry shifts away from personal car ownership towards a shared model, facial recognition will be able to lock car doors or engines until the registered user needs their ride, as well as being able to change the car settings to their preferences.

Taking the school register

“Here Miss”. The school register is an unchanging part of school life. Children have been saying this for decades. But in Australia, schools are trialing an alternative. Victoria’s Department of Education is using facial recognition to monitor the whereabouts of students. Teachers and staff can access the information on a web dashboard or mobile app to save them time. Book-lending library systems using built-in face recognition could also become common, with the system scanning the person holding the borrowed book and updating its database.

The journey to the sports stadium

DOWNLOAD THE FACIAL RECOGNITION WHITEPAPER HERE

Back to top

Who’s working on face recognition technology?

A large number of tech companies are developing facial biometrics systems. Facebook uses the technology to identify and tag the millions of photos its users post every day. It says its ‘DeepFace’ tech has a true positive rate of 97.25%. Google developed a similar product called FaceNet. It claimed 99.63% accuracy when matching 13,000 pictures of faces from across the web. Amazon has developed a cloud-based facial recognition service called Rekognition, and Microsoft has a similar tech: Face API. Both companies license their tech to third parties.

Gemalto’s Cogent Live Face Identification System recognizes faces even in dynamic, uncontrolled environments. Its SDK lets developers create applications that match live faces with faces from documents. In tests, Cogent Live Face achieved a face acquisition rate of 99.44% in less than five seconds.

One promising application of this tech is in financial services. Xavier Larduinat, Head of Marketing for Digital Banking and Payment at Gemalto’s parent company Thales, says facial identification could invigorate high street banking.

“Banks don’t want to close branches. They want to reinvent them,” he says. “We’re working on kiosk systems that work 24 hours a day. They will let people scan a physical document, such as a passport, and then take a ‘liveness detection’ selfie. The tech will then compare the two images, confirm a match and process the transaction.”

Larduinat believes that the concept is secure too. “The system doesn’t store any image data here. It just makes a real-time, one-to-one comparison. This concept might also reduce the vulnerability presented by the human agent, who can pose a security risk – there have been cases of tellers that work with criminals, for example.”

DOWNLOAD THE FACIAL RECOGNITION WHITEPAPER HERE

Back to top

Facial recognition and face identification: what are the security issues?

The two big tests of any authentication system are accuracy and security. Facial recognition is undoubtedly achieving accuracy levels to rival any other biometric technique. But can new facial recognition systems defend themselves against attackers ‘stealing’ people’s faces?

As we’ve established, a face is not a ‘secret’. On the contrary, it is easy to find a person’s face in an era of photo sharing. Instead, a face has to be hard to copy – and probably ‘live’.

In the past, criminals have tried methods such as:

Photographs, 3D-printed masks and video clips

Early systems were vulnerable to these approaches. In 2009, authorized hackers successfully used photos to trick the systems used by Lenovo, Asus and Toshiba laptops.

Forced or unaware facial recognition

If attackers cannot spoof a system, they might try to force an authentication. For example, they could hold a person’s phone to the owner’s face when asleep, or coerce them to unlock it.

Stealing the numerical code

If every face pattern is converted into a numerical code before matching, then a criminal could just try to steal the codes.

Happily, technology improvements and a smarter approach to the user interface have made it harder for hackers.

Liveness detection is the best defense against photo spoofing. 3D scanning helps (see technology section) and some systems require subjects to blink during set-up to indicate their ‘liveness’.

Another sensible measure is to support different levels of security depending on the use case. In low-risk scenarios, facial recognition alone might be suitable. But where the risk is high, the system might demand multi-factor authentication such as password and fingerprint.

How is facial recognition evolving? Dimitrios Pavlakis, ABI Research

Dimitrios Pavlakis, Industry Analyst at ABI Research, says the user experience should depend on the context.

“It might be fine to wake up your in-car entertainment with just your face, but maybe not to unlock the car itself. Manufacturers and algorithm developers might also ship face recognition systems with variable thresholds for different use cases and applications. They might dial down the accuracy and increase the authentication speed in smart homes, for example, where there are a small number of frequent users, as opposed to banking or access control where multiple users need to be identified.”

Apple is a case in point here. While a person’s face alone can unlock the iPhone, a face scan and PIN code is required to unlock more sensitive services.

A good defense against the harvesting of facial data is to avoid keeping it in a central database. This is why many systems store the numerical code locally – inside a secure enclave in the device itself.

In a phone, for example, an embedded secure element (eSE) is a tamper-proof chip that lives in the chipset or SIM card. It can only be accessed with strong authentication. Also, the eSE never shares the code with an application. Instead, if a service wants to verify the user is authentic, it simply receives a yes/no answer.

Facial recognition: trust and the role of regulation

Many people worry about the use of facial recognition – especially for surveillance. This is understandable. When a system misidentifies a person, he or she could be wrongly detained by law enforcement. And if a system is worse at identifying particular groups than others (perhaps because of insufficient training data), people belonging to these groups will be more likely to fall victim to wrongful identification.

There are fundamental questions of privacy, consent and function creep. How can people be sure their facial data is not being shared? What protections are there for people that are of no interest to the police? Could ‘potential’ criminals be detained before committing a crime?

These concerns have been long recognized within the industry. The Biometrics Institute was set up in 2001 to promote the responsible and ethical use of biometrics. “Often, legislation can’t keep up,” explains Chief Executive Isabelle Moeller. “The technology is moving so fast that it’s very difficult to provide the right framework in time.”

With standards taking a long time to develop, many companies recognize the need to test their systems rigorously. Meanwhile bodies such as the Biometrics Institute are working hard to help organizations grapple with the big questions. “What does it mean if we collect people’s data? How do we store that information securely? These are the types of questions we’re seeking to answer on a day-to-day basis,” says Moeller.

In 2019, the Institute updated its privacy guidelines to factor in the growth of artificial intelligence, drones and more sophisticated facial recognition systems. It continues to build on the work being done by organizations across the board. In 2018, Microsoft published an important post.

It said regulation is needed to prevent tech companies alone having to choose between “social responsibility and market success”.

It shared six guiding principles:

1. Fairness

Facial recognition technology should treat all people fairly.

2. Transparency

Tech companies should document the capabilities and limitations of the technology.

3. Accountability

There should be an appropriate level of human control for uses that may affect people in consequential ways.

4. Non-discrimination

Terms of service should prohibit unlawful discrimination.

5. Notice and consent

Companies should provide notice and secure consent when they deploy facial recognition.

6. Lawful surveillance

There should be safeguards for people’s democratic freedoms in law enforcement surveillance scenarios.

In some countries, there is already regulation that tackles some of these issues.

The European Union introduced the General Data Protection Regulation (GDPR) on 25 May 2018. GDPR puts measures in place to limit the ways in which enterprises can gather, store and share personal data.

It classifies facial data as a ‘special category’ of data because it reveals racial or ethnic origin, genetics, biometrics and more. As such, it prohibits processing it unless an exemption applies. One of these exemptions is consent (specific, informed and unambiguous). However, the regulation does not define consent in great detail, and has yet to be tested thoroughly.

In public places, where the tech is used for surveillance or targeted advertising, it is hard to see how consent can be explicit. A venue might display a sign explaining that facial recognition will be used. But is consent ‘freely given’ in this scenario? Can anyone opt out? What if a person enters without seeing the sign?

Industry insiders believe that transparency is absolutely necessary to the ethical development of facial recognition – and that regulation, such as GDPR, provides it. ABI Research’s Dimitrios Pavlakis says: “Data protection is vital for face recognition, citizens need to know how their data is being used. Innovation and technological progress should not exclude responsibility."

Frederic Trojani, chairman of the Security Identity Alliance, agrees with Pavlakis: “The way biometric data will be used should be explicitly explained to the people. Regulations need to set clear rules on individual privacy and data protection. People need answers to questions such as: will my data be stored? For what reason? For how long? Do I have the right to erase it? GDPR is a good example of how to do this.”

The Security Identity Alliance published a set of best practices and recommendations around civil liberties and facial recognition in June 2019.

It should also be noted that GDPR is not restricted to EU-based companies. It covers the processing of personal data belonging to EU citizens – and therefore applies to organizations based in other continents.

In the US, the federal government has not introduced regulation on facial recognition. However, many states are considering the issue.

In May 2019, San Francisco’s Board of Supervisors voted by eight to one to ban the use of the technology by local agencies, such as law enforcement. The move was seen as a gesture, because the police department does not currently deploy facial identification.

Nevertheless, industry insiders believe regulators should reserve judgement before outlawing the tech outright. Joseph Hoellerer, Senior Manager, Government Relations, at the Security Industry Association, says: “We view any moves to ban facial recognition as premature and problematic.

“Law makers need to look at the issue holistically. They need the full picture before acting. The best way to characterize face recognition in law enforcement is to see it as one of many available tools. It should not be used as the sole basis for apprehending someone. But it should not be rejected either.”

Melissa Doval, CEO of Kairos, agrees there are important ethical questions to be addressed around facial recognition. US-based Kairos offers technology that companies can use to apply facial identification to their own databases. Developers can use Kairos’s APIs to match the same face or even detect whether a face is present.

Does facial recognition have an ethics issue? Melissa Doval, Kairos

Used ethically, facial recognition can improve citizen security by helping the police to detect and catch criminals faster. It might also help prevent crime before it has occurred. But regulations need to be put in place to limit its use to well-identified and legitimate cases. Completely banning this technology seems hasty, which is why cities such as London have established expert panels to examine the issues.

Doval believes any debate should consider factors beyond the technology alone. “These are human questions,” she says. “Facial recognition systems are not like connected cars, for example, where the vehicle itself might need to make a potentially life or death decision. Facial recognition merely feeds the information that a person acts on.”

She is confident that society will eventually settle on an ethical framework for the tech. But in the meantime, Kairos is selective about its client base. It turns away customers on a daily basis, and works with those that apply facial recognition to matching/authentication and not surveillance.

DOWNLOAD THE FACIAL RECOGNITION WHITEPAPER HERE

Back to top