Using Computer Vision to Play Minesweeper

As a side project I have created a small Computer Vision framework that continuously captures the screen output of a computer and provides a collection of functions to automate office tasks and play existing computer games. Among the more complicated applications is a script that automates flying an airplane in Battlefield Vietnam by detecting the horizon line in every frame.

Here I would like to showcase a rather simple application on the famous Minesweeper game. It uses pattern matching (normalized two-dimensional cross-correlation) to find and identify the cells in Minesweeper. It then selects the most likely cell not to include a bomb and performs a mouse click on it. The algorithm to solve Minesweeper could be improved using a thorough probabilistic framework or Convolutional Neural Networks, but they were not the main focus of this study.
To speedup the system we only take a screenshot every few seconds and therefore the game state can defer from what we assume it is. This is the cause for the most obvious mistakes made in the video.