Python Mouse Movement Emulation In Games

I'm looking into using the Kinect to play video games. This requires the emulation of both keyboard and mouse events. I've figured out that keyboard events with ctypes' SendInput()

Solution 1:

You can install the PyAutoGUI GUI automation module from PyPI (run pip install pyautogui) and then call the pyautogui.moveTo() to click on a certain X and Y coordinates of the screen:

>>>import pyautogui>>>pyautogui.moveTo(50, 100)

Behind the scenes on Windows, it makes similar ctypes code you are using, so it might work to control the camera in a game like Skyrim.

PyAutoGUI works on Windows, Mac, and Linux, and on Python 2 and 3. It also can emulate the keyboard, do mouse drags, take screenshots, and do simple image recognition of the screenshots. Full docs are at https://pyautogui.readthedocs.org/

Solution 2:

I have fixed your code and it's working for me in win10. Please see the changes and try.

This will move the cursor relative to the current position

import ctypes
import time

# C struct redefinitions
PUL = ctypes.POINTER(ctypes.c_ulong)

classKeyBdInput(ctypes.Structure):
    _fields_ = [("wVk", ctypes.c_ushort),
                ("wScan", ctypes.c_ushort),
                ("dwFlags", ctypes.c_ulong),
                ("time", ctypes.c_ulong),
                ("dwExtraInfo", PUL)]


classHardwareInput(ctypes.Structure):
    _fields_ = [("uMsg", ctypes.c_ulong),
                ("wParamL", ctypes.c_short),
                ("wParamH", ctypes.c_ushort)]


classMouseInput(ctypes.Structure):
    _fields_ = [("dx", ctypes.c_long),
                ("dy", ctypes.c_long),
                ("mouseData", ctypes.c_ulong),
                ("dwFlags", ctypes.c_ulong),
                ("time", ctypes.c_ulong),
                ("dwExtraInfo", PUL)]


classPOINT(ctypes.Structure):
        _fields_ = [("x", ctypes.c_ulong),
                    ("y", ctypes.c_ulong)]


classInput_I(ctypes.Union):
    _fields_ = [("ki", KeyBdInput),
                 ("mi", MouseInput),
                 ("hi", HardwareInput)]


classInput(ctypes.Structure):
    _fields_ = [("type", ctypes.c_ulong),
                ("ii", Input_I)]

# Actuals FunctionsdefPressKey(hexKeyCode):
    extra = ctypes.c_ulong(0)
    ii_ = Input_I()
    ii_.ki = KeyBdInput(0, hexKeyCode, 0x0008, 0, ctypes.pointer(extra))
    x = Input(ctypes.c_ulong(1), ii_)
    ctypes.windll.user32.SendInput(1, ctypes.pointer(x), ctypes.sizeof(x))


defReleaseKey(hexKeyCode):
    extra = ctypes.c_ulong(0)
    ii_ = Input_I()
    ii_.ki = KeyBdInput(0, hexKeyCode, 0x0008 | 0x0002, 0, ctypes.pointer(extra))  # lint:ok
    x = Input(ctypes.c_ulong(1), ii_)
    ctypes.windll.user32.SendInput(1, ctypes.pointer(x), ctypes.sizeof(x))

defMoveMouse(x, y):
    extra = ctypes.c_ulong(0)
    ii_ = Input_I()
    #x = int(x*(65536/ctypes.windll.user32.GetSystemMetrics(0))+1)#y = int(y*(65536/ctypes.windll.user32.GetSystemMetrics(1))+1)
    ii_.mi = MouseInput(x, y, 0, 0x0001, 0, ctypes.pointer(extra))
    cmd = Input(ctypes.c_ulong(0), ii_)
    ctypes.windll.user32.SendInput(1, ctypes.pointer(cmd), ctypes.sizeof(cmd))

defmain():
    #mouse = Mouse()
    MoveMouse(1,15)
    time.sleep(3)
    MoveMouse(1,-15)
    time.sleep(3)
    MoveMouse(1,15)
    time.sleep(3)
    MoveMouse(2,15)

main()

But you can also use ctypes.windll.user32.SetCursorPos(x,y) to move directly to a pixel position.

This one will move the cursor to the pixel position in the x/y axis on the screen

import ctypes

# Move cursor to x 500 and y 500
ctypes.windll.user32.SetCursorPos(500,500)