Voice and Language Models¶
Speaking to a robot in a corridor is the capability with the shortest path to a delighted customer and the shortest path to a serious problem. This page describes how it is wired, who owns which half, and what a partner may build against today.
For IT, technicians and integration partners
Technical reference for IT departments, URG technicians and trained partners. It is not written as operating instructions for end users.
1. Three parts, three owners¶
| Part | What it does | Where it runs |
|---|---|---|
| The shell | Owns the microphone and the loudspeaker. Records while a button is held, plays the answer, shows a state chip. | The Android kiosk application on the tablet |
| The assistant service | Wake word, context, phrase commands, the address speech is sent to, and the health of the whole chain. | A service on the site, reachable over HTTP |
| The operator surface | Offers the tile that starts and stops listening, and shows whether the chain is configured at all. | The uGo server and its browser view |
The split is not an accident of history. The microphone belongs to the shell because Android gives the permission to an application, not to a web page. The context belongs to the service because it is what the robot says, and that must be changeable without shipping a new app. The tile belongs to the surface because that is where a resident touches the robot.
2. The service interface¶
The assistant service is a NestJS application and publishes its own specification at
/docs-json. The groups relevant to an integrator:
| Route | Method | Purpose |
|---|---|---|
/api/chatbot/config |
GET, PUT | Configuration of the assistant |
/api/chatbot/context |
PUT | The system context — what the robot knows about where it is and whom it speaks to |
/api/chatbot/address |
GET, PUT | Where captured speech is sent for recognition |
/api/chatbot/commands |
POST | Register a phrase that triggers an action |
/api/chatbot/commands/{phrase} |
DELETE | Remove one |
/api/wake-word/config |
GET, PUT | The wake word |
/api/intercommunicator/* |
— | The speech path itself |
/api/health, /api/ready |
GET | Liveness and readiness |
MEASURED 27.08.2026 against a running service.
3. How the operator surface reaches it¶
Through our own server, at /api/chatbot/**, and never directly from the browser. The
reasons are the same three that govern every proxy in this system: the browser must not hold
the address of a device, the address belongs in one place in the runtime configuration, and
the surface must know only its own server.
Two restrictions on that proxy are worth stating, because they are what makes it safe to have at all:
It requires a technician sign-in. Whoever changes the system context changes what the robot says to residents. That is not an operator action.
The speech path is blocked, deliberately. communicate and intercommunicator return
403 through our proxy. They belong to the shell, which owns the microphone. Two paths for
voice data would be two paths somebody has to audit, and the second one would exist only for
convenience.
4. The seam to the shell¶
The tile in the operator surface does not record anything. It asks the shell to start and to stop, and it asks for the state to display. The shell offers three methods under one name; the browser side is a thin wrapper with its own tests.
| Method | Behaves like |
|---|---|
| start listening | pressing and holding the shell's own microphone button |
| stop listening | releasing it |
| state | one of unconfigured, listening, capturing, sending, playing, cooldown, error |
The tile only draws itself when the shell offers the bridge. If it does not, nothing appears. This is worth a sentence because the alternative is worse than a missing feature: a resident who presses a button and gets no reaction concludes the robot is broken, and tells everybody so.
State as of 27.08.2026
The bridge is built and tested on the browser side. On the shell side it is not yet present, so today the tile stays hidden. Nothing further is needed from the surface once it appears.
4.1 Voice command with confirmation (development state 23.10.2026)¶
Status: implemented on development branches of the shell and the surface, not yet released, tested against the chassis simulation, not yet on a device. Operator and technician view: Voice command with confirmation.
The microphone bubble of the shell in the lower right corner (round button plus state chip) stays the control for talking; the surface does not record. What the surface adds is the question before anything moves: a keyword of a destination opens "Drive to …?" on the screen, and only a touch on Yes creates the drive order, over the same drive path as a destination chosen by hand. For this the shell offers five methods to the surface (a sixth and a seventh one, for the technician page and the web-service view, are described under Embedded pages below):
| Method | Direction | Meaning |
|---|---|---|
| state | shell → surface | one of unconfigured, listening, capturing, sending, playing, cooldown, error |
| latest event | shell → surface | the last utterance as a small JSON object with a running number: understood with the transcript, not understood, or fault (recognition service not reachable) |
| language | surface → shell | the language shown on the surface: de, en, fr, es, it or nl |
| movement | surface → shell | while true, the microphone bubble is hidden and no new recording starts |
| persona name | surface → shell | name of the assistant after the chatbot persona; shown in front of the state text, plain text, at most 40 characters |
The surface polls once per second. An event that was already present when the page loaded is never acted on.
Where the decision is made. The keywords are kept in the uGo server (see
uGo server, /api/sprachziele) and matched by the
surface. The assistant service only has to deliver the transcript; it does not decide about a
destination and must not create a drive order of its own.
Language. With each recorded chunk the shell sends the language chosen on the surface as
the additional form field language (two letters, as above). Whether the assistant service
uses the field for recognition is to be confirmed with the operator of that service.
Only the uGo page in the main frame. The web view offers the bridge to every frame, also to an embedded web service, and the shell cannot tell which frame calls. Therefore the shell draws a new random frame key on every page load of the main frame and hands it over only to the uGo page in the main frame, after loading. Every method takes this key as its first value and answers only while the main frame shows the configured start address (same scheme, host and port); without both a call gets an empty answer and has no effect. A refused call is logged without its content.
What the shell writes to its log. Neither transcripts nor audio, only the kind of an event and its length.
Embedded pages and the robot network. Mixed content is never allowed in the web view: a web service that runs as https cannot call plain-http addresses such as the chassis or the u-IoT module. In addition the shell refuses every plain-http request of the web view to a private, loopback or link-local address other than its start address. This second rule has known limits: it does not see WebSocket connections, a public host name that resolves to a private address, or the target of a redirect. For a web service that runs as plain http, the mixed-content rule does not apply; on the web-service surfaces the shell therefore sets its own limits (see below), and the separated network is a hard precondition. The protected network segment stays a precondition in every case. Development state, not yet checked on a device.
One exception: the u-IoT console on the technician page. The technician page shows the u-IoT
module's own interface (plain http, private address) in a frame. Mixed content does not apply,
because the uGo page is plain http as well. Before the frame is shown, the page reads the u-IoT
addresses saved in uGo — the u-IoT address of the settings and the module addresses of the
Building tab, as origins, at most 64 (/api/uiotfreigabe, technician only) — and hands them to
the shell with a sixth bridge method, rahmenFreigabeSetzen, guarded by the frame key like the
other five; closing the frame hands over an empty list. The shell lets such a request through
only if its origin is on that list, it comes from a frame and not the main frame, the main frame
shows the technician page of the start address, and a referrer, if the web view reports one, is
the start address or a listed origin. A new page load and leaving the technician page end the
exception. The web-service view lives on other pages of the surface and never gets it; public,
https, loopback and link-local entries are not taken. The shell takes the hand-over on any page of
the uGo surface, because the bridge call can arrive before the web view has reported the page
change; it applies the list only on the technician page. Whether the web view reports the
referrer to the shell is not yet checked on a device.
A second, separate exception: saved web services in the web-service view. Web services the
technician saves on the technician page are meant to load on both surfaces of the web-service
view — the page opened by the tile Start web services and the view on the follow screen — also
when they run as plain http on a private address of the customer network; for plain http only an
address as a number is accepted, no name (a name may be a second name of the u-IoT module or the
robot computer). Right before the view shows a service, the page reads the origins of the saved,
active http web services (/api/webdienstfreigabe; addresses of the robot and its modules —
chassis, u-IoT, building modules, mobile base, voice services, the robot computer — are always
left out, compared by address after name resolution) and hands the origin of the opened service,
and only that one, to the shell with a seventh bridge method, webdienstFreigabeSetzen, guarded
by the frame key; closing the view hands over an empty list. The shell answers how many origins
it took; if it took none, the page tries once more and then shows not reachable instead of an
empty frame. The shell keeps this list apart from the u-IoT list:
it applies only while the main frame shows one of the two web-service surfaces, never on the
technician page, and the u-IoT list never applies in the web-service view. The other conditions
are the same as above (frame only, referrer check, a new page load and leaving both surfaces end
it); entries on the robot computer itself are not taken. Mixed content stays forbidden: if the
surface itself were https, an http web service would not load, and the view says so instead of
showing an empty frame. Development state, not yet checked on a device.
Limits the shell sets on the web-service surfaces. While the main frame shows one of the two web-service surfaces, the shell decides every request itself:
- plain http passes only to the start address and to the handed-over origin of the opened service; every other plain-http address — private or public, address or name — is refused;
- the shell loads GET and HEAD requests to that origin itself. A redirect to another origin is refused and its target never contacted; a redirect inside the origin is followed. Every answer gets an additional content security policy that allows connections (fetch, XHR, WebSocket) only to the service's own origin, no nested frames other than its own origin and https, no workers from elsewhere, no form submission and no plug-ins;
- other methods to that origin are sent by the web view, limited by that policy;
- forms are not submitted in the web-service view, cookies are not kept.
Consequences for a plain-http web service: it loads data only from its own address (scheme, host, port); a service that needs a second address, a redirect to another address or a form login does not work embedded — save its final https address instead. Not limited by the shell: WebRTC. Whether the web view applies the policy to the answers the shell loads is not yet checked on a device. Therefore plain http is allowed only where the customer network is separated from the robot network: the chassis and the u-IoT module must not be reachable from the customer network. The commissioning test includes a step that checks this.
A security note that belongs with it. Exposing an interface to a web view opens the application to every page that view loads. It is acceptable here only because the shell restricts itself to the host of its start address. That restriction has to stay; without it the bridge becomes a way in.
5. Where speech actually goes — and why this page says no more¶
Captured audio leaves the robot to be recognised. Where it goes, over which protocol, and what happens to it afterwards are questions with answers, and those answers are being worked through with the operator of that service before the first customer deployment.
This page does not print them. Not because they are embarrassing, but because an interface description is the wrong place for the current state of somebody's transport security: a finding gets fixed and then described, not published while it is open. What a partner needs from us here is the shape of the chain, and that is above.
What is fixed, and what a data protection officer can be told today:
- Speech is captured only while a button is held. There is no always-on microphone in this design, and the wake word is a separate, configurable feature that is off unless a site asks for it.
- The robot works in full without any of this. A site that does not want a voice assistant simply has no address configured, and the tile never appears.
6. Open¶
| Question | State |
|---|---|
| Recognition service: operator, location, processing agreement | OPEN — to be settled before the first customer deployment |
| Retention of recordings after recognition | OPEN |
| The bridge on the shell side | Implemented on a development branch (23.10.2026), not yet released, only for the uGo page in the main frame (frame key) — see 4.1 |
| Assistant service creates no drive order of its own, only the transcript | OPEN — to be confirmed before voice control is switched on at a site |
Use of the language field by the assistant service |
OPEN |
| Multilingual assistant answers | The surface carries six languages; the assistant's own language handling is not yet described |