Bridging the gap

Here’s the important stuff up front:

To git the repo of just the mouse emulator code for use with any web application, go to this link.

Access the mouse build of JSketcher demonstrating touchpad functionality, at this link.

Introduction

In Computer-Aided Design (CAD), precision and intricate interactions are paramount. Enter JSketcher, a CAD application tailored for the desktop environment, leveraging the full spectrum of mouse events for drawing geometry, selecting edges, and manipulating models. From the simple click to the more nuanced mouseover and mousemove events, and even the right-click and scroll wheel actions, JSketcher is a masterclass in rich mouse interactions.

However, as the digital landscape evolved, so did the need for mobility. The challenge? Adapting JSketcher’s intricate desktop-centric interactions for the mobile platform. Given the application’s deep-rooted reliance on mouse events, a straightforward modification was off the table. The complexity of these interactions made a simple mobile adaptation not just challenging but near impossible without overhauling the core mechanics of the application.

While one might assume that such challenges are commonplace in software development, my search for a touchpad simulator in JavaScript yielded no existing solutions. This gap in the market sparked an idea: what if I could craft a generic tool that overlays existing applications, simulating a mouse cursor and firing all the events typically triggered by a traditional pointing device?

In this article, I’ll share my journey in bridging the gap between desktop and mobile for JSketcher, introducing a virtual mouse touchpad that revolutionizes mobile interactions for intricate web applications.

So, What’s the Problem in Detail?

In the world of CAD, precision is paramount, and the tools used must offer intricate interactions to meet the demands of the craft. JSketcher, a browser-based CAD application, was meticulously designed with this in mind. Primarily tailored for desktop use, it uses an extensive array of mouse interactions to function at its best. Let’s delve deeper into the intricacies of the problem:

Rich mouse interactions: JSketcher wasn’t just about simple clicks. It utilized a comprehensive range of mouse events:

• Click events: The foundational interactions for selecting and drawing.
• Mouse move: Essential for tracking cursor movement, especially when drawing or dragging elements.
• Right-click: Often used for context-specific actions or opening context menus.
• Scroll wheel: For zooming in and out of models or scrolling through options.
• Mouseover and Mousemove: Crucial for highlighting elements when the cursor hovers over them, offering real-time feedback.
• Mousein and Mouseout: These events detect when the mouse enters or leaves a specific element, respectively, often used to trigger visual feedback or additional interactions.
• Pointerin and Pointerout: Similar to mousein and mouseout but designed to handle mouse and touch events, providing a more unified event model.

Mobile limitations: Mobile devices present a unique set of challenges:

• Absence of a right-click.
• Traditional hover actions, like mouseover, don’t exist on touch devices.
• The need to find an alternative for the scroll wheel action, given mobile devices lack one.
• Handling events like mousein, mouseout, pointerin, and pointerout, which are more nuanced on touch devices.

The core challenge: The primary obstacle wasn’t merely about translating desktop mouse actions to mobile touch actions. It was about retaining the intricate user experience that JSketcher users had come to depend on. A mere tap on the screen couldn’t replicate the multifaceted actions a mouse could execute. Moreover, a complete redesign of the mobile application wasn’t a viable option, considering the intricate nature of the application’s architecture.

Searching for Existing Solutions

When faced with the challenge of adapting JSketcher for mobile, the path wasn’t immediately clear. The application’s deep-rooted reliance on many mouse events meant that traditional mobile adaptation methods might not suffice. Here’s a look at some initial approaches and why they fell short:

1. Simple touch-to-click mapping: The most straightforward solution would be to map touch events to mouse clicks. A tap could simulate a click. A swipe could emulate mouse movement, and so on. However, this approach was too simplistic. It didn’t account for the rich set of interactions JSketcher offered, especially nuanced events like mouseover, mouseout, pointerin, and pointerout.

2. Mobile-first redesign: Another consideration was redesigning the application with a mobile-first approach. This would involve reimagining the user interface and interactions for touch devices. But given JSketcher’s complexity of the existing design, this would be a massive undertaking.

3. External plugins or libraries: The JavaScript ecosystem is vast, and there are plugins and libraries for almost everything. I explored whether existing tools could bridge the gap between desktop and mobile interactions. While some offered partial solutions, none could fully replicate the intricate mouse events JSketcher relied on.

4. Hybrid approach: Combining touch events with on-screen buttons or controls was another option. For instance, on-screen buttons could simulate right-clicks or scroll wheel actions. However, this could clutter the interface and still not address all the challenges.

Despite these initial attempts, it became evident that a more innovative solution was needed. One that could seamlessly integrate with JSketcher without compromising its rich interaction capabilities.

My Solution and Adventure in Triggering Mouse Events

Navigating the challenges of adapting JSketcher for mobile required a fresh perspective. The solution I envisioned was not just about simulating mouse events but creating an interactive layer that would bridge the gap between touch and mouse interactions. Enter the Virtual Mouse Touchpad.

The Virtual Mouse Touchpad architecture:

• Separate HTML page: The touchpad was designed as a standalone HTML page. This modular approach ensured it could be easily integrated with various applications, not just JSketcher.
• Iframe integration: The desktop application, in this case, JSketcher, is displayed in the background using an iframe. This allows the application to run in its native environment while the touchpad interface interacts.
• Transparent UI elements: Overlaying the iframe were transparent UI elements for the virtual touchpad and mouse buttons. These elements had minimalist borders, providing intuitive guides for users to know where to place their fingers.
• The virtual pointer: More than just a static image, the virtual pointer is dynamic. It is an image with its absolute location set by the virtual mouse, reflecting the exact position of the virtual mouse cursor in real time.

This layered approach ensured that users had a seamless experience. They could view and interact with the desktop application through the iframe, while the virtual mouse touchpad provided the necessary mouse-like interactions. The goal was clear: to offer the richness of desktop interactions on mobile devices without compromising the user experience.

Explain the Code

The concept of the Virtual Mouse Touchpad, while clear in design, required intricate coding to bring to life. Here’s a breakdown of the key components:

1. Transparent UI elements: The virtual touchpad and mouse buttons were designed as transparent HTML elements with minimalist borders.

<meta content='width=device-width; initial-scale=1.0; maximum-scale=1.0; user-scalable=0;' name='viewport' />


<style>
    #holderHolder {
        margin: 0;
        border: 0;
        padding: 0;
    }

    #FakeMouseHolder {
        position: fixed;
        bottom: 0;
        left: 0;
        width: 100%;
        height: 100%;
        z-index: 2;
        display: grid;
        grid-template-columns: repeat(8, 1fr);
        grid-template-rows: repeat(7, 1fr);
        grid-column-gap: 0px;
        grid-row-gap: 0px;
        gap: 0px;
    }

    #FakeMouseHolder>div {
        border: 2px dashed rgba(0, 0, 0, 0.4);
        border-radius: 10px;
        margin: 1px;
    }


    #touchpadArea {
        grid-area: 1 / 1 / 7 / 8;
    }

    #EscButton {
        grid-area: 5 / 8 / 5 / 9;
        background-color: rgba(100, 94, 94, 0.2);
    }

    #ShiftButton {
        grid-area: 6 / 8 / 6 / 9;
        background-color: rgba(100, 94, 94, 0.4);
    }



    #leftMouseButton {
        grid-area: 7 / 1 / 8 / 5;
        background-color: rgba(100, 94, 94, 0.4);
    }


    #rightMouseButton {
        grid-area: 7 / 5 / 8 / 8;
        background-color: rgba(100, 94, 94, 0.4);
    }

    #settingsButton {
        grid-area: 7 / 8 / 8 / 9;
        background-color: rgba(100, 94, 94, 0.2);
    }


    #settings {
        top: 0px;
        bottom: 0px;
        left: 0px;
        right: 0px;
        background-color: rgb(95, 101, 101);
        position: absolute;
        z-index: 200;
        border-radius: 30px;
        border-color: darkslategray;
        border-style: solid;
        border-width: thick;
        margin: 5px;
        padding: 5px;
        text-align: left;
    }



    #mouseToggle {
        position: fixed;
        bottom: calc(30% - 20px);
        left: 0;
        width: 40px;
        height: 40px;
        background-color: black;
        border-top-right-radius: 15%;
        border-bottom-right-radius: 15%;
        border: 3px solid #73AD21;
        z-index: 100;
    }

    body {
        margin: 0;
        padding: 0;
        border: 0px;
        overflow-x: hidden;
        overflow-y: hidden;
        overflow: hidden;

    }

    iframe {
        width: 100%;
        height: 100%;
        margin: 0;
        padding: 0;
        border: 0px;
        z-index: 1;
        overflow: hidden;
        zoom: 0.25;

    }


    #cursor {
        pointer-events: none;
        position: absolute;
        transform: translate(-50%, -50%) scale(40);

    }

    input {
        width: calc(100% - 2 * 20px);
        margin: 20px;
        float: left;
    }



    #saveSettings {
        width: 100%;
        height: 100%;
        font-size: large;
        cursor: pointer;
        text-align: center;
        text-decoration: none;
        background-color: #4CAF50;
    }
</style>

<body>
    <iframe id="pointerTarget" scrolling="no"></iframe>

    <div id="holderHolder">
        <div id="FakeMouseHolder">
            <div id="touchpadArea"></div>
            <div id="EscButton">ESC</div>
            <div id="ShiftButton">shift</div>

            <div id="leftMouseButton" class="mousebutton">🔓</div>
            <div id="rightMouseButton" class="mousebutton">🔓</div>
            <div id="settingsButton">settings</div>
        </div>
    </div>


    <button id="mouseToggle" onclick="toggleMousepad()">🖱</button>
    <style>
        #settings {
            display: grid;
            grid-template-columns: repeat(3, 1fr);
            grid-template-rows: repeat(8, 1fr);
            grid-column-gap: 0px;
            grid-row-gap: 0px;
            margin: 10%;
        }

        #settings>div {
            border: 2px dashed rgba(0, 0, 0, 0.4);
            border-radius: 10px;
            margin: 0px;
            overflow: hidden;
        }

        #settingsWindowLabel1 {
            grid-area: 1 / 1 / 2 / 2;
        }

        #settingsWindow1 {
            grid-area: 1 / 2 / 2 / 6;
        }

        #settingsWindowLabel2 {
            grid-area: 2 / 1 / 3 / 2;
        }

        #settingsWindow2 {
            grid-area: 2 / 2 / 3 / 6;
        }
    </style>

    <div id="settings">
        <div id="settingsWindowLabel1"></div>
        <div id="settingsWindow1">
            <button id="saveSettings">Close Settings</button>
        </div>

        <div id="settingsWindowLabel2">Mouse Speed</div>
        <div id="settingsWindow2">
            <input id="mouseSpeed" type="range" value=".6" min="0.1" max="2" step="0.1"
                onchange="this.nextElementSibling.value = this.value">
            <output>.6</output>
        </div>
    </div>

</body>

2. The iframe integration: To seamlessly integrate JSketcher within the touchpad interface, an iframe was employed.

const pointerTarget = document.getElementById("pointerTarget");

3. Dynamic virtual pointer: The position of the virtual pointer was dynamically updated based on user interactions with the touchpad.

document.body.innerHTML += `
  <svg id="cursor" xmlns="http://www.w3.org/2000/svg" viewBox="-10003 -10003 20010 20010">
  <path d="M 0 0 L 0 10000 Z M 0 0 L 0 -10000 M 0 0 L -10000 0 M 0 0 L 10000 0 M 25 0 A 1 1 0 0 0 -25 0 A 1 1 0 0 0 25 0" stroke="black" stroke-width="3" fill="none"/>
  </svg>`;

cursor = document.getElementById("cursor");
cursor.style.width = "100px";
cursor.style.height = "100px";

4. Toggle functionality: A button was introduced to let users alternate between the virtual touchpad and direct iframe interaction.

const FakeMouseHolderDiv = document.getElementById("FakeMouseHolder");
const holderHolder = document.getElementById("holderHolder");

function toggleMousepad(showHide) {
  if (showHide) {
    if (showHide == "show") holderHolder.style.display = "block";
    if (showHide == "hide") holderHolder.style.display = "none";
    return;
  }

  if (holderHolder.style.display == "none") {
    holderHolder.style.display = "block";
  } else {
    holderHolder.style.display = "none";
  }
}

5. Event simulation: The core of the touchpad’s functionality is simulating mouse events based on touchpad interactions. For this, we need to use the virtual mouse position and get the element at that x, y. You can use the elementsFromPoint() function to accomplish this. This is part of the script injected into the iframe.

The bubble attribute of the event is important. If you have a div with an onclick event that contains a child element like a picture or text, the child element will be returned when we execute elementFromPoint(). Also, the bubbles option allows us to trigger the proper events on each parent of the particular dom element.

Here’s a simplified example of sending a click event to a particular X, Y position:

const itemUnderMouse = document.elementFromPoint(absoluteX, absoluteY);


eventToSend = {
  type:"click",
  bubbles: true,
  cancelable: false,
  view: window,
  screenX: absoluteX,
  screenY: absoluteY,
};

eventToSend = new MouseEvent(eventToSend.type, eventToSend);
itemUnderMouse.dispatchEvent(eventToSend);

This code demonstrates how multiple events can be triggered based on the type of mouse interaction. These are not always 1:1 with the event types used and must trigger all events expected for a particular mouse action.

For example, mousemove really needs to trigger mousemove and mouseover. Click needs to trigger click, mousedown, and mouseup. It gets complicated real quick. We also need to keep track of any elements we have done a mouseover and mouse enter action on to trigger mouseout and pointerout events when those items are no longer under the cursor.

Below is an example of the logic that triggers multiple event types based on more generic mouse actions:

function doTheProperEvents(item) {
  if (eventType == "mousemove") {
     exicuteEvents(item, ["mousemove", "mouseover"]);

    if (!mouseOverList.includes(item)) {
      mouseOverList.push(item);
       exicuteEvents(item, ["mouseenter", "pointerenter"]);
    }
  }

  if (eventType == "leftDragStart") {
    exicuteEvents(item, ["click", "mousedown", "pointerdown"]);
  }

  if (eventType == "leftDragEnd") {
    exicuteEvents(item, ["mouseup", "pointerup"]);
  }

  if (eventType == "rightDragStart") {
    exicuteEvents(item, ["contextmenu","auxclick", "mousedown", "pointerdown"], { button: 2 });
  }

  if (eventType == "rightDragEnd") {
    exicuteEvents(item, ["mouseup", "pointerup"], { button: 2 });
  }

  if (eventType == "click") {
    if (item.nodeName == "INPUT") item.focus();

    exicuteEvents(item, ["click", "mousedown", "mouseup"]);
  }
  if (eventType == "dblclick") {
    if (item.nodeName == "INPUT") item.focus();

    exicuteEvents(item, ["click", "mousedown", "mouseup", "dblclick"]);
  }
  if (eventType == "rightclick") {
    exicuteEvents(item, ["contextmenu", "auxclick"]);
    exicuteEvents(item, ["click", "mousedown", "mouseup", "pointerdown", "pointerup"], { button: 2 });
  }
} 


function exicuteEvent(TargetElement, eventToSend = {}) {
  eventToSend.clientX = absoluteX;
  eventToSend.clientY = absoluteY;
  eventToSend.x = absoluteX;
  eventToSend.y = absoluteY;
  eventToSend.pageX = absoluteX;
  eventToSend.pageY = absoluteY;
  eventToSend.view = window;
  eventToSend.bubbles = true;
  eventToSend.cancelable = true;
  eventToSend.shiftKey = shiftKey;

  eventToSend = new MouseEvent(eventToSend.type, eventToSend);
  try {
    testResult =  TargetElement.dispatchEvent(eventToSend);
    if (mouseDebugger) if (!testResult) console.log("event trigger failed", testResult, TargetElement, eventToSend);
    //if (TargetElement.dispatchEvent(eventToSend) == false) console.log("event trigger failed", TargetElement, eventToSend);
    return testResult;
  } catch {
    if (mouseDebugger) console.log("event trigger failed", TargetElement, eventToSend);
    return "failed";
  }
}

 function exicuteEvents(TargetElement, eventTypes, eventToSend = {}) {
  const eventTemplate = JSON.parse(JSON.stringify(eventToSend));
  eventTypes.forEach( (enenvtToFire, key) => {
    eventTemplate.type = enenvtToFire;
     exicuteEvent(TargetElement, eventTemplate);
  });
}

7. Message passing between the iframe and Parent: Communication between the iframe (JSketcher) and its parent (the Virtual Mouse Touchpad) was crucial for the seamless functioning of the system. This was achieved using message passing.

Why message passing? Given the sandboxed nature of iframes, direct interactions between the parent and the iframe can be restricted. Message passing provides a secure and effective way to communicate between the two.

Here’s what the inside of the iframe looks like:

window.addEventListener(
  "message",
   (event) => {
    const thingToDo = event.data;
    // handle doing events for the message......
  },
  true
);

The code in the iframe parent:

obj = {
  leftMouseDown: false,
  rightMouseDown: false,
  absoluteX: 100,
  absoluteY: 100,
  deltaY: 0,
  shiftKey: false,
};
pointerTarget.contentWindow.postMessage(obj);

8. Injecting a JavaScript module into the iframe: To further enhance the functionality and integration of the Virtual Mouse Touchpad, injecting JavaScript directly into the iframe was necessary. The injected code listens for messages passed from the parent and triggers the events inside the iframe.

The injection process: injecting a module involves creating a script element, setting its source to the desired module, and appending it to the iframe’s document.

pointerTarget.onload = function () {
  const elem = document.createElement(`script`);
  elem.src = "./mouse/virtualMousePointer.js";
  elem.type="module";

  pointerTarget.contentDocument.body.appendChild(elem);

  console.log(window.location);
};

document.body.onload = function () {
  try {
    document.getElementById("mouseSpeed").value = localStorage.mouseSpeed;
  } catch {}
  pointerTarget.src = "./" + window.location.search;
  //console.log("dats the window locations", window.location.search);
};

The Results

After hours of coding, testing, and refining, the Virtual Mouse Touchpad was finally ready to debut. The results were nothing short of transformative for JSketcher’s mobile experience.

1. Seamless integration: The touchpad, being a separate HTML page overlaying JSketcher through an iframe, ensured that the core application remained untouched. This meant that updates or changes to JSketcher wouldn’t affect the touchpad’s functionality.
2. Intuitive user experience: With transparent UI elements and a dynamic virtual pointer, users could easily understand and navigate using the touchpad. The minimalist borders provided clear guidance, and the toggle functionality allowed users to switch between interaction modes effortlessly.
3. Comprehensive mouse event simulation: From basic clicks to intricate events like mouseover, mouseout, pointerin, and pointerout, the touchpad successfully replicated the full spectrum of mouse interactions on mobile devices.

Making a Desktop Web App Mobile-Friendly With a Virtual Mouse Touchpad was originally published in Better Programming on Medium, where people are continuing the conversation by highlighting and responding to this story.