Mr. Moritz Blum Profile

Moritz Blum

at Diebold Nixdorf Inc

SPIE Involvement:

Author

Publications (1)

Proceedings Article | 11 October 2018 Presentation

A practical approach for optical skimming and automatic identification of the pressed contents on touchscreen devices (Conference Presentation)

Dinh Khoi Le, Volker Krummel, Moritz Blum, Benedikt Lueke

Proceedings Volume 10802, 108020P (2018) https://doi.org/10.1117/12.2325347

KEYWORDS: Mobile devices, Video, Cameras, Image processing, Machine vision, Computer vision technology, Detection and tracking algorithms, Cell phones, Instrument modeling, Information operations

Read Abstract +

Entering confidential information is the classical application for optical skimming. For example entering a PIN or password to authenticate banking transactions is one of the most sensitive moments of a transaction. Up to now it was unclear how efficient optical skimming attacks can be mounted even on mobile devices. In this paper we show that filming the entering of a password with the camera of a standard mobile phone is enough for a fully automated recovering of the sensitive information. Our analysis method of the recorded video stream leads to a success rate of more than 90%. In our model a user enters his password into an Android or IOS driven touchscreen device while being filmed by the attacker’s smartphone. The goal of the attacker is to derive the password from the movie instantly, e.g. to use it in a real time man-in-the-middle attack. On first sight such an attack seems hard to mount due to many disturbing factors like movements of the device, bad light conditions etc. However, we show that many of these disturbing factors can be mitigated by smart video analytics. We implemented the whole attack in our lab simulating real conditions. Parts of the setup where a Samsung S7 for filming and an iPhone 6 as target. For real time processing we use the computer vision library OpenCV which supports most of the common image and video processing algorithms. Our goal is to cover most of the important cases while holding as little knowledge as possible about the video the algorithm has to work on. Starting from movable or instable position for camera and device we apply a discrete Fourier transformation to obtain a constant plan view as starting point for our algorithm. To obtain the necessary information for this the user has to select the four corners of the keyboard which is observed by hand. In real live situations the device is hold by hand of the user so we decided to use the MOSSE algorithm for stabilization to deal with the problem of a jiggling device. The keystroke detection is based on the “pop up” feature of common smartphone keyboards. This event is remarkable enough to get detected and located by simple gray value subtraction of following frames. Not only the magnified keys are the primary changes we observed and analyzed. Minor changes like noise can be removed by sufficiently blurring the frame. Subtracting areas where normal skin colors are detected yields information about the position of the keystrokes. The relative position obtains a probability for the pressed keys by analyzing the activity of the regions where the “pop ups” usually are. Our work showed that optical skimming of passwords are practical even on mobile devices. We show that it is possible to make a simple attack with basic computer vision techniques. Up to date mobile devices have enough computing power to provide for this kind of attacks. We give recommendations for entering passwords securely.

View contact details

UPDATE YOUR PROFILE

Is this your profile? Update it now.

Sign into your SPIE.org account

Don’t have a profile and want one?

Create an account on SPIE.org

Keywords/Phrases

Search In:

Publication Years