Bluetooth Mouse 2 Keyboard [updated]

Based on BLEMouse2XAC BLEmouse2xac project – BLE Mouse/Trackball to Xbox Adaptive Controller Joystick

What it is

This project allows a BLE 5Bluetooth Low energy) mouse movement to be converted in keyboard arrow movements. An ESP32 (M5STack Atom S3) is connected to the BLE mouse via blueooth, and to a mobile phone (or a PC) via an USB link. The ESP 32 is seen as a keyboard that sue keycodes corresponding to the direction arrows.

(as we use an ESP32-S3, we are not compatible with bluetooth 2.0/ Bluetooth classic. The porject in its current state only support BLE mouses. The nimBLE library is also BLE only)

It is meant to be used with with the upcoming “Design for All” open source library for Android made by Orange S.A.

The primary use is a disabled person using a (BLE) wheelchair joystick. The wheelchair joystick , connected to a mobile phone through the ESP32 device running EST32Mouse2Keyboard can drive efficiently the TASO app (Telephone adapté aux Salariés d’Orange) and other apps using the DfA (DEsign for all) library.

usage during tests - the AtomS3 is connected to a mac for debugging

usage during tests - close up

Hardware

The ESP32-S3 is powered throught the usb hub connected to the android phone.

The ESP32-S3 does not have bluetooth 2.0 (Classic) support. Previous version of the ESP 32 has support.

M5Stack AtomS3

This device is available with and without a built-in display. Set the IDE board to “M5SAtomS3”.

To put the board in upload mode, press and hold the small button on the side for about 2 seconds until a green LED lights up inside. Start uploading.

Power cycle the board after the upload is done. Might be nessary to do it 3 times).

Arduino macOS IDE options to build the project for the AtomS3

Arduino IDE (macOS) options to build the project for the AtomS3. USBmode:USB OTG (tiny USB), USB CDC on boot:Disabled.

IDE options to build the project for the AtomS3

IDE options to build the project for the AtomS3 - P, as from project BLEMOuse2XAC.

Arduino configuration

ESP 32 :

Arduino Board definition (as time of editing)

M5Stack

You should have the M5AtomS3 definition.

If you use M5STack-AtomS3, it woill not compile.

Software

Install the following libraries using the Arduino IDE library manager.

  • OneButton by Matthias Hertel, 250
  • NimBLE-Arduino by h2zero, 1.4.1
  • M5GFX by M5Stack 0.1.13
  • ESP32TinyUSB by Darius Stempa, 2.0.2

OPtionnaly, you can install FastLed by Daniel Garcia 3.6.0 or later installed, as some dead code use it.

IMPORTANT ! : Install M5-Stack Board definitions as explained in the web page https://docs.m5stack.com/en/arduino/arduino_board .

To do that, copy the M5Stack board management URL below into the Additional Board Manager URLs: field, and save.

https://static-cdn.m5stack.com/resource/arduino/package_m5stack_index.json

THen select M5AtomS3 as board.

Compiler Warnings

Ignore the warning messages about “WM_NOTEMP” and “No hardware SPI pins defined”.

In file included from /Users/henry/Documents/Arduino/libraries/FastLED/src/FastLED.h:75,
                 from /Users/henry/Documents/Arduino/libraries/M5AtomS3/src/./utility/LedDisplay.h:4,
                 from /Users/henry/Documents/Arduino/libraries/M5AtomS3/src/M5AtomS3.h:6,
                 from /Users/henry/Documents/Arduino/M5atomS3blemouse/M5atomS3blemouse.ino:31:
/Users/henry/Documents/Arduino/libraries/FastLED/src/fastspi.h:157:23: note: #pragma message: No hardware SPI pins defined.  All SPI access will default to bitbanged output
 #      pragma message "No hardware SPI pins defined.  All SPI access will default to bitbanged output"
                       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Display Messages

“Scanning”

Waiting for a BLE mouse or trackball. The first time a mouse/trackball is used, it must be paired. See the device instructions for pairing. It usually is done by holding a button down until an LED starts blinking.

“Mouse to XAC”

The program is converting mouse/trackball buttons and movement to joystick buttons and movements.

“Disconnect Scanning”

The mouse/trackball will go into a power save mode when idle which disconnects BLE. Pressing a button should wake up the device and reconnect without pairing.

Mouse to keyboard conversion

Information can be received every 30 ms from the mouse. We must filter values. A finite state automata process the data. (Not a very ‘pure’ one : it has one variable (which could be seen as a stack of depth 1) and also count the time).

Finite state automata for recognizing input and apply repeat delay and rate.

Notes

When not working, press side button (reset) : happens always on power-up

When powered/reprogrammed, the program often block the first time, shows a messed up display on second reboot, and works perfectly on third reboot (and further on).

Use the board definition named M5AtomS3, not the booard definition named ‘M5Stack-AtomS3’

Wireless Mouse/Trackball

Not all wireless mice and trackballs support BLE. Some use the older Bluetooth Classic protocols. Some use proprietary (not Bluetooth or WiFi) wireless protocols. The ESP32-S3 only works with BLE devices.

Important

On Arduino IDE, select M5AtomS3 as board. Do not use the M5Stack-AtomS3 board provided by other environments (esp32 by Expressif ?). See Software section above for detailed instructions.

Amongst other things, USB debugging won’t work.

USB Debug

Debug output on USB enabled

“USB CDC On Boot:” “Enabled” #define USB_DEBUG 1

Debug output on USB disabled

“USB CDC On Boot:” “Disabled” #define USB_DEBUG 0

Project on gitlab: https://gitlab.com/hmt63/esp32mouse2keyboard

Further reading