Skip to content

Voice and Language Models

Speaking to a robot in a corridor is the capability with the shortest path to a delighted customer and the shortest path to a serious problem. This page describes how it is wired, who owns which half, and what a partner may build against today.

For IT, technicians and integration partners

Technical reference for IT departments, URG technicians and trained partners. It is not written as operating instructions for end users.

1. Three parts, three owners

Part What it does Where it runs
The shell Owns the microphone and the loudspeaker. Records while a button is held, plays the answer, shows a state chip. The Android kiosk application on the tablet
The assistant service Wake word, context, phrase commands, the address speech is sent to, and the health of the whole chain. A service on the site, reachable over HTTP
The operator surface Offers the tile that starts and stops listening, and shows whether the chain is configured at all. The uGo server and its browser view

The split is not an accident of history. The microphone belongs to the shell because Android gives the permission to an application, not to a web page. The context belongs to the service because it is what the robot says, and that must be changeable without shipping a new app. The tile belongs to the surface because that is where a resident touches the robot.

2. The service interface

The assistant service is a NestJS application and publishes its own specification at /docs-json. The groups relevant to an integrator:

Route Method Purpose
/api/chatbot/config GET, PUT Configuration of the assistant
/api/chatbot/context PUT The system context — what the robot knows about where it is and whom it speaks to
/api/chatbot/address GET, PUT Where captured speech is sent for recognition
/api/chatbot/commands POST Register a phrase that triggers an action
/api/chatbot/commands/{phrase} DELETE Remove one
/api/wake-word/config GET, PUT The wake word
/api/intercommunicator/* — The speech path itself
/api/health, /api/ready GET Liveness and readiness

MEASURED 27.08.2026 against a running service.

3. How the operator surface reaches it

Through our own server, at /api/chatbot/**, and never directly from the browser. The reasons are the same three that govern every proxy in this system: the browser must not hold the address of a device, the address belongs in one place in the runtime configuration, and the surface must know only its own server.

Two restrictions on that proxy are worth stating, because they are what makes it safe to have at all:

It requires a technician sign-in. Whoever changes the system context changes what the robot says to residents. That is not an operator action.

The speech path is blocked, deliberately. communicate and intercommunicator return 403 through our proxy. They belong to the shell, which owns the microphone. Two paths for voice data would be two paths somebody has to audit, and the second one would exist only for convenience.

4. The seam to the shell

The tile in the operator surface does not record anything. It asks the shell to start and to stop, and it asks for the state to display. The shell offers three methods under one name; the browser side is a thin wrapper with its own tests.

Method Behaves like
start listening pressing and holding the shell's own microphone button
stop listening releasing it
state one of unconfigured, listening, capturing, sending, playing, cooldown, error

The tile only draws itself when the shell offers the bridge. If it does not, nothing appears. This is worth a sentence because the alternative is worse than a missing feature: a resident who presses a button and gets no reaction concludes the robot is broken, and tells everybody so.

State as of 27.08.2026

The bridge is built and tested on the browser side. On the shell side it is not yet present, so today the tile stays hidden. Nothing further is needed from the surface once it appears.

4.1 Voice command with confirmation (development state 23.10.2026)

Status: implemented on development branches of the shell and the surface, not yet released, tested against the chassis simulation, not yet on a device. Operator and technician view: Voice command with confirmation.

The microphone bubble of the shell in the lower right corner (round button plus state chip) stays the control for talking; the surface does not record. What the surface adds is the question before anything moves: a keyword of a destination opens "Drive to …?" on the screen, and only a touch on Yes creates the drive order, over the same drive path as a destination chosen by hand. For this the shell offers five methods to the surface (a sixth and a seventh one, for the technician page and the web-service view, are described under Embedded pages below):

Method Direction Meaning
state shell → surface one of unconfigured, listening, capturing, sending, playing, cooldown, error
latest event shell → surface the last utterance as a small JSON object with a running number: understood with the transcript, not understood, or fault (recognition service not reachable)
language surface → shell the language shown on the surface: de, en, fr, es, it or nl
movement surface → shell while true, the microphone bubble is hidden and no new recording starts
persona name surface → shell name of the assistant after the chatbot persona; shown in front of the state text, plain text, at most 40 characters

The surface polls once per second. An event that was already present when the page loaded is never acted on.

Where the decision is made. The keywords are kept in the uGo server (see uGo server, /api/sprachziele) and matched by the surface. The assistant service only has to deliver the transcript; it does not decide about a destination and must not create a drive order of its own.

Language. With each recorded chunk the shell sends the language chosen on the surface as the additional form field language (two letters, as above). Whether the assistant service uses the field for recognition is to be confirmed with the operator of that service.

Only the uGo page in the main frame. The web view offers the bridge to every frame, also to an embedded web service, and the shell cannot tell which frame calls. Therefore the shell draws a new random frame key on every page load of the main frame and hands it over only to the uGo page in the main frame, after loading. Every method takes this key as its first value and answers only while the main frame shows the configured start address (same scheme, host and port); without both a call gets an empty answer and has no effect. A refused call is logged without its content.

What the shell writes to its log. Neither transcripts nor audio, only the kind of an event and its length.

Embedded pages and the robot network. Mixed content is never allowed in the web view: a web service that runs as https cannot call plain-http addresses such as the chassis or the u-IoT module. In addition the shell refuses every plain-http request of the web view to a private, loopback or link-local address other than its start address. This second rule has known limits: it does not see WebSocket connections, a public host name that resolves to a private address, or the target of a redirect. For a web service that runs as plain http, the mixed-content rule does not apply; on the web-service surfaces the shell therefore sets its own limits (see below), and the separated network is a hard precondition. The protected network segment stays a precondition in every case. Development state, not yet checked on a device.

One exception: the u-IoT console on the technician page. The technician page shows the u-IoT module's own interface (plain http, private address) in a frame. Mixed content does not apply, because the uGo page is plain http as well. Before the frame is shown, the page reads the u-IoT addresses saved in uGo — the u-IoT address of the settings and the module addresses of the Building tab, as origins, at most 64 (/api/uiotfreigabe, technician only) — and hands them to the shell with a sixth bridge method, rahmenFreigabeSetzen, guarded by the frame key like the other five; closing the frame hands over an empty list. The shell lets such a request through only if its origin is on that list, it comes from a frame and not the main frame, the main frame shows the technician page of the start address, and a referrer, if the web view reports one, is the start address or a listed origin. A new page load and leaving the technician page end the exception. The web-service view lives on other pages of the surface and never gets it; public, https, loopback and link-local entries are not taken. The shell takes the hand-over on any page of the uGo surface, because the bridge call can arrive before the web view has reported the page change; it applies the list only on the technician page. Whether the web view reports the referrer to the shell is not yet checked on a device.

A second, separate exception: saved web services in the web-service view. Web services the technician saves on the technician page are meant to load on both surfaces of the web-service view — the page opened by the tile Start web services and the view on the follow screen — also when they run as plain http on a private address of the customer network; for plain http only an address as a number is accepted, no name (a name may be a second name of the u-IoT module or the robot computer). Right before the view shows a service, the page reads the origins of the saved, active http web services (/api/webdienstfreigabe; addresses of the robot and its modules — chassis, u-IoT, building modules, mobile base, voice services, the robot computer — are always left out, compared by address after name resolution) and hands the origin of the opened service, and only that one, to the shell with a seventh bridge method, webdienstFreigabeSetzen, guarded by the frame key; closing the view hands over an empty list. The shell answers how many origins it took; if it took none, the page tries once more and then shows not reachable instead of an empty frame. The shell keeps this list apart from the u-IoT list: it applies only while the main frame shows one of the two web-service surfaces, never on the technician page, and the u-IoT list never applies in the web-service view. The other conditions are the same as above (frame only, referrer check, a new page load and leaving both surfaces end it); entries on the robot computer itself are not taken. Mixed content stays forbidden: if the surface itself were https, an http web service would not load, and the view says so instead of showing an empty frame. Development state, not yet checked on a device.

Limits the shell sets on the web-service surfaces. While the main frame shows one of the two web-service surfaces, the shell decides every request itself:

  • plain http passes only to the start address and to the handed-over origin of the opened service; every other plain-http address — private or public, address or name — is refused;
  • the shell loads GET and HEAD requests to that origin itself. A redirect to another origin is refused and its target never contacted; a redirect inside the origin is followed. Every answer gets an additional content security policy that allows connections (fetch, XHR, WebSocket) only to the service's own origin, no nested frames other than its own origin and https, no workers from elsewhere, no form submission and no plug-ins;
  • other methods to that origin are sent by the web view, limited by that policy;
  • forms are not submitted in the web-service view, cookies are not kept.

Consequences for a plain-http web service: it loads data only from its own address (scheme, host, port); a service that needs a second address, a redirect to another address or a form login does not work embedded — save its final https address instead. Not limited by the shell: WebRTC. Whether the web view applies the policy to the answers the shell loads is not yet checked on a device. Therefore plain http is allowed only where the customer network is separated from the robot network: the chassis and the u-IoT module must not be reachable from the customer network. The commissioning test includes a step that checks this.

A security note that belongs with it. Exposing an interface to a web view opens the application to every page that view loads. It is acceptable here only because the shell restricts itself to the host of its start address. That restriction has to stay; without it the bridge becomes a way in.

5. Where speech actually goes — and why this page says no more

Captured audio leaves the robot to be recognised. Where it goes, over which protocol, and what happens to it afterwards are questions with answers, and those answers are being worked through with the operator of that service before the first customer deployment.

This page does not print them. Not because they are embarrassing, but because an interface description is the wrong place for the current state of somebody's transport security: a finding gets fixed and then described, not published while it is open. What a partner needs from us here is the shape of the chain, and that is above.

What is fixed, and what a data protection officer can be told today:

  • Speech is captured only while a button is held. There is no always-on microphone in this design, and the wake word is a separate, configurable feature that is off unless a site asks for it.
  • The robot works in full without any of this. A site that does not want a voice assistant simply has no address configured, and the tile never appears.

6. Open

Question State
Recognition service: operator, location, processing agreement OPEN — to be settled before the first customer deployment
Retention of recordings after recognition OPEN
The bridge on the shell side Implemented on a development branch (23.10.2026), not yet released, only for the uGo page in the main frame (frame key) — see 4.1
Assistant service creates no drive order of its own, only the transcript OPEN — to be confirmed before voice control is switched on at a site
Use of the language field by the assistant service OPEN
Multilingual assistant answers The surface carries six languages; the assistant's own language handling is not yet described