pointer move and click via xdotool with WebXR example

mouse move with command with options, with regex verification
kwin embedding
2023-10-25 10:28:18 +02:00 · 2023-10-24 22:11:45 +02:00 · 2023-10-23 18:16:06 +02:00 · 2023-10-23 18:04:14 +02:00 · 2023-10-23 17:57:58 +02:00 · 2023-10-23 10:50:48 +02:00
5 changed files with 3344 additions and 123 deletions
--- a/README.md
+++ b/README.md
@@ -1,43 +0,0 @@
-# Offtopus, offline-octopus
-
-## Goal
-Provide a way to connect local machines to work and tinker with efficiently, building prototypes with resilience in mind.
-
-## Demo
-
-5min recorded demo video https://video.benetou.fr/w/sHE39WSZkQgfPNDdiypfog
-
-Note that XR is optional. Original extract from https://youtu.be/BRjohy0ruAg?t=2061 for Future of Text demo day.
-
-## How does it work
-It provides a Web server with endpoints for different functions, including listing machines joining that network, make functions available to that new network of machines, sharing files, etc. Note that the point is to build on top of it, consequently functions as endpoints here are solely examples.
-
-## How to install
-1. clone repository
-1. install dependancies (no package.json yet) but mostly express missing, i.e `npm i express`
-1. run e.g `node .`
-1. connect to it locally first, e.g https://localhost:8082 (assumes existing SSL certificates) or https://localhost:8082/routes
-
-Note that the cloudinit file is an example on a brand new machine.
-
-### tested on
- Linux machines, e.g Ubuntu on desktop, RPi Zero
- Android devices using Termux, e.g Quest 1
- iOS via iSH
-Most likely partially works on Windows with Linux subsystem. System commands, e.g shutdown, xrandr, etc must be adapted.
-
-### Developing and debugging
- See the console REPL, starting with help()
- see `/routes/json` to connect to all services
-
-## Digging deeper
- Future of Text WebXR demo with shared filesystems https://video.benetou.fr/w/h291CfZiezenY1t46cQQo5 extract from https://www.youtube.com/results?search_query=demo+prototype+session
- hour long video discussion https://video.benetou.fr/w/aR81WVHg6H3E93GPG4jYUg
- self hosting AI notes https://fabien.benetou.fr/Content/SelfHostingArtificialIntelligence
- self hosting subreddit https://old.reddit.com/r/selfhosted/
- small Web https://ar.al/2020/08/07/what-is-the-small-web/
- DWeb https://en.wikipedia.org/wiki/Decentralized_web
-
-### Inspiration
- Designing for serendipity: supporting end-user configuration of ubiquitous computing environments, Xerox PARC 2002
- Providing an Integrated User Experience of Networked Media, Devices, and Services through End-User Composition, 2008
--- a/16
+++ b/16
@@ -1,16 +0,0 @@
-#!/bin/bash
-
-cd /root
-curl -sL https://deb.nodesource.com/setup_20.x -o nodesource_setup.sh
-source nodesource_setup.sh 
-apt install -y nodejs
-apt update && apt install -y git
-git clone https://git.benetou.fr/utopiah/offline-octopus
-mkdir Prototypes
-mv offline-octopus/ Prototypes/
-cd Prototypes/offline-octopus/
-npm i express
-openssl req  -nodes -new -x509 -keyout naf-key.pem -out naf.pem -subj "/C=BE/ST=Brussels/L=Brussels/O=Global Security/OU=IT Department/CN=offtopus.benetou.fr"
-touch /root/.ssh/id_rsa_offlineoctopus.pub
-touch /root/Prototypes/offline-octopus/.keyfrommd5
-HOME=/root /usr/bin/node index.js &>> log.txt
--- a/examples/events.html
+++ b/examples/events.html
@@ -1,29 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-  <head>
-
-    <meta charset="utf-8" />
-    <meta name="viewport" content="width=device-width, initial-scale=1" />
-    <link rel="icon" href="https://glitch.com/favicon.ico" />
-    <title>offtopus events</title>
-
-  </head>
-  <body>
-    <div id="content"></div>
-    <hr>
-    <div id="events"></div>
-    
-<script>
-const source = new EventSource('/events');
-
-source.addEventListener('message', message => {
-  console.log('Got', message);
-
-  // Display the event data in the `content` div
-  document.querySelector('#events').innerHTML = event.data;
-})
-
-    </script>
-    
-  </body>
-</html>
--- a/examples/spasca-offline/engine/index.html
+++ b/examples/spasca-offline/engine/index.html
--- a/index.js
+++ b/index.js
@@ -8,10 +8,10 @@ const express = require("express");           // could be good to replace with c
 // (worked at least on RPi, Deck, Quest and x64 machines)

 // Get port or default to 8082
-const port = process.env.PORT || 8082;
-const protocol = 'https'
-const subclass = '10.160.168.'
-//const subclass = '192.168.4.'
+const port = process.env.PORT || 8082; // assumes homogenity across peers
+const protocol = process.env.PROTOCOL || 'https'
+const subclass = process.env.SUBCLASS || '192.168.0.' // defaulting to IP used at home rather than RPi0. Could also change it there as it's not justified beside helping distinction.
+// Object.values(require("os").networkInterfaces()).flat().filter(({ family, internal }) => family === "IPv4" && !internal).map(({ address }) => address)[0].split('.').slice(0,3).join('.')+'.'

 const publicKeyPath = path.resolve(process.env.HOME,'.ssh','id_rsa_offlineoctopus.pub')
 const publicKey = fs.readFileSync(publicKeyPath).toString().split(' ')[1]
@@ -28,6 +28,12 @@ const minfileSaveFullPath = path.join(__dirname,'examples', minfilename)
 const sshconfigpath = path.resolve(process.env.HOME,'.ssh','config')
 const propath = path.resolve(process.env.HOME,'Prototypes')

+const sshfsmounts = "sshfsmounts"
+
+let workspaces = {}
+let mountPoints = []
+// note that stopping this process removes the mounts
+
 // does not apply in a P2P fashion, must rely on local configuration here config.json
 let localServices = [ ]
 const configFilePath = path.resolve(__dirname, "config.json")
@@ -49,6 +55,17 @@ pmwiki http localhost 4000 /pmwiki.php
 sshd see quest in ssh config, specific user and port
 */

+const kwinmin = `
+echo "const clients = workspace.clientList();
+for (var i = 0; i < clients.length; i++) {
+  print(clients[i].caption);
+  clients[i].minimized = true;
+}" > /tmp/kwinscriptdemo
+
+num=$(dbus-send --print-reply --dest=org.kde.KWin /Scripting org.kde.kwin.Scripting.loadScript string:"/tmp/kwinscriptdemo" | awk 'END {print $2}' )
+
+dbus-send --print-reply --dest=org.kde.KWin /$num org.kde.kwin.Script.run`
+
 const utilsCmd = { // security risk but for now not accepting user input so safer
 	//'update' : { desc: 'note that will lose the state, e.g foundpeers', cmd: 'killall '+process.title+' && ' },
 	// should first download the new version and proceed only if new
@@ -61,6 +78,19 @@ const utilsCmd = { // security risk but for now not accepting user input so safe
 	'listprototypes': { cmd: 'ls', context: {cwd: propath},
 		format: res => res.toString().split('\n')
 	},
+	'mousemove' : { cmd: 'xdotool mousemove ', optionsPattern: /\d+ \d+/},
+	'mouseup' : { cmd: 'xdotool mouseup 1'},
+	'mousedown' : { cmd: 'xdotool mousedown 1'},
+	// per device specific (until adjustable per user)
+	'minimizeall' : { cmd: kwinmin}, //KWin script
+	//'minimizeall' : { cmd: '/home/fabien/Prototypes/kwin-scripting/launch'}, //KWin script
+	'highresscreen' : { cmd: 'xrandr --output DP-4 --mode 3840x2160'},
+	'lowresscreen' : { cmd: 'xrandr --output DP-4 --mode 1920x1080'},
+	// 'anyresscreen' : { cmd: 'xrandr --output DP-4 --mode '}, // would require user input which is risky e.g here 'OxO; wget rootkit; bash rootkit;'
+	'availableRes' : { cmd: 'xrandr -q | grep 0x | grep -v primary | cut -f 4 -d " "',
+		format: res => res.toString().split('\n').filter( l => l!='' )
+	},
+
 	//'npmfind' : { desc: 'package manager finder', cmd: 'find . -wholename "*node_modules/acorn"' }, 
 		// security risk if relying on user provided name, e.g replacing acorn by user input
 		// example that could be generalized to other package managers e.g .deb or opkg
@@ -68,6 +98,7 @@ const utilsCmd = { // security risk but for now not accepting user input so safe
 // could be interesting to consider also recent containers and ~/.bashrc for services

 const instructions = `
+<pre>
 /home/deck/.ssh/
 	trusted context, i.e on closed WiFi and over https with bearer authorization
 /home/deck/.ssh/config
@@ -89,6 +120,7 @@ ssh remarkable2 to get drawing preview
 util functions
 	modify WiFi parameters, including AP if available
 	shutdown/reboot
+</pre>
 `

 const auth_instructions = `generate md5 from pub offline-octopus then provide as bearer query param`
@@ -106,6 +138,19 @@ app.use(function(req, res, next) {

 app.get('/', (req, res) => {
 	res.send( instructions )
+	// see issue 20
+})
+
+app.get('/routes', (req, res) => {
+	let formattedRoutes  = routes.map( r => `<li><a href='${r}'>${r}</a></li>` ).join('\n')
+	res.send( formattedRoutes )
+})
+
+app.get('/routes/json', (req, res) => {
+	// minimalist for JXR but could also be OpenAPI, cf https://git.benetou.fr/utopiah/offline-octopus/issues/16
+		// used in /webxr as
+	// fetch('/routes/json').then(r=>r.json()).then(r=>r.filter(p=>p.match(/resolution/)).map(p=>addNewNote('jxr fetch("https://192.168.0.129:8082'+p+'")')))
+	res.send( routes )
 })

 app.get('/authtestviaheader', (req, res) => {
@@ -177,8 +222,8 @@ app.get('/exec', (req, res) => {
 	}
 })

-function execConfiguredCommand(cmdName){
-	let resultFromExecution = execSync(utilsCmd[cmdName].cmd, utilsCmd[cmdName].context) 
+function execConfiguredCommand(cmdName, options=''){
+	let resultFromExecution = execSync(utilsCmd[cmdName].cmd+options, utilsCmd[cmdName].context) 
 	let formatter = utilsCmd[cmdName].format
 	if (formatter) resultFromExecution = formatter(resultFromExecution)
 	return resultFromExecution
@@ -220,9 +265,11 @@ app.get('/recentfiles', (req, res) => {
 	res.json( {msg: 'not yet implemented'})
 })

-let dynURL = 'https://192.168.4.1/offline.html'
+let dynURL = '/'
 app.get('/hmdlink/set', (req, res) => { // could be a PUT instead
 	// e.g http://192.168.4.3:8082/hmdlink/set?url=http://192.168.4.3:8082/114df5f8-3921-42f0-81e7-48731b563571.thumbnails/f07120ba-0ca1-429d-869f-c704a52b7aa3.png
+	// should return a warning if req.query.url is undefined
+	if (!req.query.url) {res.json( {error: 'use /hmdlink/set?url=YOURURL'}); return}
 	dynURL = req.query.url 
 	res.redirect( dynURL )
 })
@@ -232,7 +279,8 @@ app.get('/hmdlink', (req, res) => {
 })

 app.get('/webxr', (req, res) => {
-	res.redirect( '/local-metaverse-tooling/local-aframe-test.html' )
+	res.redirect( '/spasca-offline/engine/index.html')
+	//res.redirect( '/local-metaverse-tooling/local-aframe-test.html' )
 })

 // user for /scan to populate foundPeers
@@ -251,9 +299,8 @@ app.get('/scan', (req, res) => {
 })

 function scanpeers(){
-	sendEventsToAll({'action':'scanning started'})
 	foundPeers = []
-	for (let i=1;i<25;i++){ // async so blasting, gives very quick result for positives
+	for (let i=1;i<254;i++){ // async so blasting, gives very quick result for positives
 		let url=protocol+'://'+subclass+i+':'+port+'/available'
 		let opt={rejectUnauthorized: false}
 		https.get(url, opt, res => {
@@ -274,20 +321,57 @@ function scanpeers(){

 app.get('/sshconfig', (req, res) => {
 	res.json( getSshConfig() )
-	// should filter on foundPeers to avoid offline peers
 })

-// note that stopping this process removes the mounts
+app.get('/sshconfig/live', (req, res) => {
+	res.json( getSshConfig().filter( i => foundPeers.map( e => subclass+e ).filter( x => i.connectionString.match(x) ).length ) )
+})
+
+app.get('/unifiedworkspaces', (req, res) => {
+	res.json( workspaces )
+})
+
+function getUnifiedworkspaces(){
+	mountPoints.map( mp => {
+		let dir = path.resolve(__dirname, sshfsmounts, mp)
+		workspaces[mp] = fs.readdirSync( dir )
+		console.log('reading', mp, workspaces.mp)
+	})
+	return workspaces
+}
+
 function mountAll(){
-	getSshConfig().map( l => {
+	// should scanpeers() first
+	if (foundPeers.length==0) return
+	getSshConfig().filter( i => foundPeers.map( e => subclass+e ).filter( x => i.connectionString.match(x) ).length  )
+	.map( l => {
 		let cs = 'sshfs ' + l.name + ':'
 		if (l.custom)
 			cs+= l.custom
 		else
 			cs+='/home/'+l.user
-		return cs + ' ' + path.resolve(__dirname, "sshfsmounts", l.name)
+		let targetPath = path.resolve(__dirname, sshfsmounts, l.name)
+		if (!fs.existsSync(targetPath)){ fs.mkdirSync(targetPath, { recursive: true }); }
+		mountPoints.push(l.name)
+		return cs + ' ' + targetPath
 		} )
-		.map( l => execSync(l))
+	//.map( l => console.log(l))
+	.map( l => { try { execSync(l) } catch (err) { console.log(err) } } )
+}
+
+function umountAll(){
+	mpath = path.resolve(__dirname, sshfsmounts)
+	fs.readdirSync( mpath )
+	// could rely on mountPoints instead
+	.map( f => {
+		try {
+			    execSync('umount ' + path.resolve(__dirname, sshfsmounts, f) )
+			// should update mountPoints instead
+		    } catch (err) {
+			    console.log(err)
+		    }
+	})
+	mountPoints = []
 }

 function getSshConfig(){
@@ -330,6 +414,17 @@ function getSshConfig(){
 	easier to revoke if need be
  */

+app.get('/mouse/up', (req, res) => { res.json( execConfiguredCommand('mouseup') ) })
+app.get('/mouse/down', (req, res) => { res.json( execConfiguredCommand('mousedown') ) })
+app.get('/mouse/move', (req, res) => {
+	if( !req.query.options?.match(utilsCmd['mousemove']) ) { {res.json( {error: 'use /mouse/move?options=100 2000'}); return}}
+	res.json( execConfiguredCommand('mousemove',req.query.options) ) }
+)
+app.get('/minimizeall', (req, res) => { res.json( execConfiguredCommand('minimizeall') ) })
+app.get('/resolution/high', (req, res) => { res.json( execConfiguredCommand('highresscreen') ) })
+app.get('/resolution/low', (req, res) => { res.json( execConfiguredCommand('lowresscreen') ) })
+app.get('/resolution/list', (req, res) => { res.json( execConfiguredCommand('availableRes') ) })
+
 app.get('/localprototypes', (req, res) => {
 	// examples to disentangle own work for cloned existing repositories :
 		// find Prototypes/ -iwholename */.git/config | xargs grep git.benetou.fr
@@ -372,6 +467,9 @@ app.get('/editor/save', (req, res) => {
 	}
 })

+let routes = app._router.stack.map( r => r.route?.path ).filter( r => typeof(r) == 'string' )
+// it's a form of caching but so far routes are not dynamically generated so sufficient solution
+
 const privateKey = fs.readFileSync("naf-key.pem", "utf8");
 const certificate = fs.readFileSync("naf.pem", "utf8");
 const credentials = { key: privateKey, cert: certificate };
@@ -381,30 +479,85 @@ const webServer = https.createServer(credentials, app);
 // const webServer = http.createServer(app);
 // Listen on port
 webServer.listen(port, () => {
-    console.log("listening on "+protocol+"://localhost:" + port);
+	console.log("listening on "+protocol+"://0.0.0.0:" + port)
+	getCommand()
 });

-// SSE from https://www.digitalocean.com/community/tutorials/nodejs-server-sent-events-build-realtime-app
-// adapted from jxr-permanence
-let clients = [];
+// REPL testing
+const readline = require("readline");
+const rl = readline.createInterface({
+    input: process.stdin,
+    output: process.stdout,
+    completer: completer
+});

-function eventsHandler(request, response, next) {
-  const headers = {
-    'Content-Type': 'text/event-stream',
-    'Connection': 'keep-alive',
-    'Cache-Control': 'no-cache'
-  };
-  response.writeHead(200, headers);
-  const clientId = Date.now();
-  const newClient = { id: clientId, response };
-  clients.push(newClient);
-  request.on('close', () => { clients = clients.filter(client => client.id !== clientId); });
+// https://nodejs.org/api/readline.html#use-of-the-completer-function
+function completer(line) {
+  const completions = help().split('\n');
+  const hits = completions.filter((c) => c.startsWith(line));
+  // Show all completions if none found
+  return [hits.length ? hits : completions, line];
 }

-function sendEventsToAll(data) {
-  // function used to broadcast
-  clients.forEach(client => client.response.write(`data: ${JSON.stringify(data)}\n\n`))
+let command = ''
+function getCommand(){
+	rl.question(process.title+" REPL: ", function(command) {
+	    if (command == "close") {
+		rl.close();
+	    } else {
+		    try {
+			    console.log(command, eval(command) )
+		    } catch (err) {
+			    console.log(err)
+		    }
+		    getCommand()
+		    // somehow switch to node REPL proper after?!
+	    }
+	});
 }

-app.get('/events', eventsHandler);
-// for example /events.html shows when /scan begings (but not ends)
+function help(){
+	return `
+help()
+execConfiguredCommand(cmdName)
+getSshConfig()
+mountAll()
+umountAll()
+scanpeers()
+getUnifiedworkspaces()
+foundPeers
+port
+protocol
+subclass
+publicKeyPath
+publicKey
+md5fromPub
+process.title
+filename
+fileSaveFullPath
+minfilename
+minfileSaveFullPath
+sshconfigpath
+propath
+localServices
+configFilePath
+utilsCmd
+instructions
+auth_instructions
+process.title
+sshfsmounts
+mountPoints
+workspaces
+`
+}
+
+rl.on("close", function() {
+    console.log("\ndone");
+    umountAll()
+    process.exit(0);
+});
+
+// Demo Day target :
+// show files from ~/Prototypes as cubes from ssh mounted on a virtual workspace
+	// sshfs might not even be needed, see allpeers/exec instead
+		// wouldn't easily get content back though, just meta data
Author	SHA1	Message	Date
Utopiah	9414807070	pointer move and click via xdotool with WebXR example	2023-10-25 10:28:18 +02:00
Utopiah	c6e72023fc	mouse move with command with options, with regex verification	2023-10-24 22:11:45 +02:00
Utopiah	78229f9b8f	kwin embedding	2023-10-23 18:16:06 +02:00
Utopiah	8c89994e8d	WebXR example including KWin example	2023-10-23 18:04:14 +02:00
Utopiah	4af9a219c3	routes availability, WebXR route example and KWin example	2023-10-23 17:57:58 +02:00
Utopiah	ff1454d061	list all routes in HTML to make them clickable	2023-10-23 10:50:48 +02:00
Utopiah	6cb7a8c8c8	res change example	2023-10-23 10:05:47 +02:00
Utopiah	9892f29a1d	unified workspace	2023-05-12 16:48:06 +02:00
Utopiah	386416094b	REPL completion	2023-05-12 11:29:22 +02:00
Utopiah	28c5b31e2d	safer REPL	2023-05-12 11:06:37 +02:00
Utopiah	fa3ee1fc2a	basic REPL	2023-05-12 11:02:19 +02:00