Skip to main content

Chapter 13: The Living Building Interface

"The building became visible. Not just data on screens, but a breathing, thinking organism you could see and touch."

Date: October 12-13, 2025 Milestone: UI Phase 2 & 3 Complete Status: Making autonomy visible and trustworthy

The Visibility Problem​

We had built something extraordinary:

  • 🤖 Intelligent agents making autonomous decisions
  • 🛡️ OPA policies enforcing safety guardrails
  • 🏢 Multi-vendor coordination across Schneider, Avigilon, EcoStruxure
  • ⚡ Energy optimization saving real money
  • 🔒 Security orchestration preventing threats

But there was a fundamental problem: Nobody could see it working.

The agents were thinking. The policies were protecting. The building was optimizing itself. But from the outside, it was all invisible. Just logs streaming by in a terminal, traces in Jaeger, metrics in Prometheus.

How do you trust an autonomous system you can't see?

This is the story of making autonomy visible, beautiful, and trustworthy.

Part I: Phase 2 - Building the Command Centers​

The Challenge: Breaking Down Complexity​

We had three major domains:

  1. Security - Access control, surveillance, threat detection
  2. Energy - HVAC optimization, demand response, cost reduction
  3. Orchestration - Multi-agent coordination, conflict resolution

Each needed its own specialized interface. Each had unique requirements. But they all needed to work together seamlessly.

Security Command Center​

Component: SecurityCommandCenter.tsx (src/CitadelMesh.Web/src/components/dashboards/)

The first dashboard we built. A comprehensive view of building security:

Real-Time Monitoring:

  • 🚪 Door Status - All access points at a glance (locked/unlocked/unknown)
  • 👁️ Camera Feeds - Live surveillance grid (12 cameras)
  • 🚨 Active Incidents - Real-time threat tracking with severity
  • 📊 Event Timeline - Minute-by-minute activity log

Policy Transparency:

  • 🛡️ Policy Status - OPA evaluation results for every action
  • ✅ Approval Chain - See exactly why actions were allowed/denied
  • 📋 Audit Trail - Complete history of security decisions

Agent Awareness:

  • 🤖 Agent Intent - See what the security agent is thinking
  • 🎯 Current Action - Watch autonomous decisions in real-time
  • 📈 Performance Metrics - Agent response times and efficiency

Energy Operations Center​

Component: EnergyOperationsCenter.tsx (src/CitadelMesh.Web/src/components/dashboards/)

The optimization dashboard. Making complex algorithms human-readable:

Zone Management:

  • 🌡️ Temperature Grid - All HVAC zones with current/target temps
  • 🎯 Setpoint Control - Visual sliders with OPA policy limits
  • 👥 Occupancy Tracking - Real-time people counting
  • ⚠️ Status Indicators - Normal/warning/alert color coding

Cost & Savings:

  • 💰 Real-Time Cost - Current energy spend ($/hour)
  • 📊 Savings Today - Money saved by optimization
  • ⚡ Power Usage - kWh tracking across zones
  • 📈 Historical Trends - 24-hour cost/savings graphs

Grid Integration:

  • 🔌 Demand Response - OpenADR 2.0b event participation
  • ⚡ Peak Shaving - Load reduction during high-cost periods
  • 🌱 Carbon Impact - CO₂ reduction metrics

Building Orchestrator Dashboard​

Component: BuildingOrchestrator.tsx (src/CitadelMesh.Web/src/components/dashboards/)

The coordination view. Multi-agent symphony visualization:

Active Workflows:

  • 🔄 Running Tasks - All active agent operations
  • 🎯 Priority Queue - What's being worked on (Safety > Security > Comfort > Cost)
  • ⏱️ Execution Time - Real-time progress bars
  • ✅ Completion Status - Success/failure tracking

Conflict Resolution:

  • ⚔️ Active Conflicts - When agents disagree
  • 🎲 Resolution Strategy - How the orchestrator decided
  • 📊 Priority Hierarchy - Visual priority tree
  • 👤 Human Escalation - Unresolvable conflicts flagged

System Health:

  • 🟢 Agent Status - All agents online/offline
  • 📡 Communication - Event bus health
  • 🛡️ Policy Engine - OPA response times
  • 🔐 Identity - SPIRE certificate status

The Gateway Bridge​

Component: Gateway (src/CitadelMesh.Gateway/)

The critical piece connecting UI to backend:

What It Does:

// BFF (Backend For Frontend) pattern
GET  /api/security/doors        → Schneider SSE via MCP
GET  /api/security/cameras      → Avigilon ACC via MCP
GET  /api/energy/zones          → EcoStruxure EBO via MCP
POST /api/policy/explain        → OPA evaluation endpoint
WS   /ws/events                 → CloudEvents stream (NATS)

Why It Matters:

  • 🔒 Security - UI never talks directly to vendor systems
  • 🎯 Simplicity - Complex orchestration hidden from frontend
  • ⚡ Performance - Aggregates multiple backend calls
  • 🛡️ Policy Enforcement - All actions pre-checked by OPA

Part II: Phase 3 - The Digital Twin Awakens​

The Leap to 3D​

Dashboards were great. But buildings are spatial. You can't understand a building from flat panels.

We needed to see the building as it really is: a 3D space with zones, assets, and flows.

This required stepping into the world of:

  • three.js - 3D rendering engine
  • React Three Fiber - React renderer for 3D
  • @react-three/drei - Helper library for common 3D patterns

Digital Twin Spatial View​

Component: DigitalTwinSpatialView.tsx (src/CitadelMesh.Web/src/components/twin/)

The centerpiece. A living 3D model of the building:

3D Foundation:

<Canvas shadows camera={{ position: [20, 20, 20], fov: 50 }}>
  <PerspectiveCamera />
  <OrbitControls />
  <ambientLight intensity={0.5} />
  <directionalLight castShadow position={[10, 10, 5]} />
  <Environment preset="city" />
  <Grid infiniteGrid />
  <GizmoHelper />
</Canvas>

What You See:

  • 🏢 Building Shell - Wireframe box showing structure
  • 📐 Grid System - Spatial reference for navigation
  • 💡 Lighting - Ambient + directional + environment
  • 🎯 Gizmo - Orientation reference (X/Y/Z axes)
  • 👁️ Camera Controls - Orbit, zoom, pan with mouse

Performance:

  • ⚡ 60fps on mid-tier laptop
  • 🎨 Shadow mapping at 2048x2048
  • 🔄 Frustum culling for efficiency
  • 📦 Bundle size - 1.73MB (508KB gzipped)

Zone Overlays​

Component: ZoneOverlay.tsx (src/CitadelMesh.Web/src/components/twin/)

Making invisible zones visible:

Visual Representation:

// Each zone is a 3D plane with:
- Position in space [x, y, z]
- Size dimensions [width, height, depth]
- Status color (normal 🟢 / warning 🟡 / alert 🔴)
- Transparency for spatial context

Real-Time Telemetry:

  • 🌡️ Temperature - Live readings overlaid in 3D
  • 👥 Occupancy - People count with threshold alerts
  • ⚠️ Status - Visual pulse animation for alerts
  • 📊 Spheres - 3D indicators for high occupancy

Interactions:

  • Hover - Zone highlights, cursor changes
  • Click - Zone selects, displays info panel
  • Animation - Selected zones pulse up/down
  • Deselect - Click again to clear

Example Zones:

  • Main Lobby (22.5°C, 12 people, normal)
  • Office Wing A (23.1°C, 8 people, normal)
  • Office Wing B (24.8°C, 15 people, warning)
  • Server Room (26.2°C, 2 people, alert) 🔴

Asset Markers​

Component: AssetMarker.tsx (src/CitadelMesh.Web/src/components/twin/)

Every IoT device gets a 3D presence:

Asset Types & Geometries:

  • 🌡️ HVAC - Box geometry (blue) - Air handling units
  • 🚪 Door - Thin box (green) - Access control points
  • 👁️ Camera - Cone + sphere (purple) - Surveillance
  • 📡 Sensor - Sphere (pink) - IoT temperature sensors

Status Indicators:

// Visual rings at asset base:
🟢 Online     → Green glow + subtle pulse
🔴 Offline    → Gray + red pulsing ring
🟡 Maintenance → Yellow/orange indicator

Interactions:

  • Hover - Asset highlights, tooltip appears
  • Click - Asset selects, rotates continuously
  • Visual - Status ring glows, offline assets pulse
  • Info Panel - Name, type, status, zone displayed

Floor Selector​

Component: FloorSelector.tsx (src/CitadelMesh.Web/src/components/twin/)

Multi-level building navigation:

Floor Navigation:

  • 📊 Floor List - All levels with statistics
  • 🔢 Zone Count - Zones per floor
  • 🔌 Asset Count - Devices per floor
  • 🚨 Alert Badges - Red badges for active incidents

Example Building:

🏢 Four-Story Office Building
├─ Ground Floor (4 zones, 12 assets, 1 alert)
├─ Second Floor (6 zones, 18 assets, 0 alerts)
├─ Third Floor (6 zones, 15 assets, 2 alerts)
└─ Rooftop (2 zones, 8 assets, 0 alerts)

UX:

  • Click to expand/collapse
  • Active floor highlighted blue
  • Quick stats summary
  • Smooth floor transitions

Part III: Integration & Mock Data Strategy​

The Mock-First Approach​

We built the UI before the backend was ready. How?

Mock Data Structure:

// Realistic data that matches future API shape
interface Zone {
  id: string
  name: string
  position: [number, number, number]
  size: [number, number, number]
  color: string
  temperature?: number
  occupancy?: number
  status: 'normal' | 'warning' | 'alert'
}

interface Asset {
  id: string
  name: string
  type: 'hvac' | 'door' | 'camera' | 'sensor'
  position: [number, number, number]
  status: 'online' | 'offline' | 'maintenance'
  zone: string
}

Why This Works:

  1. UI development unblocked - No backend dependency
  2. API contract defined - Backend knows what to build
  3. Easy swap - Replace mock with real API calls later
  4. Realistic testing - Mock data mimics production scenarios

WebSocket Stream (Ready for Live)​

Current Implementation:

// Mock CloudEvents for development
const mockEvents = [
  { type: 'door.unlocked', door: 'Main Entry' },
  { type: 'temperature.changed', zone: 'Lobby', temp: 22.5 },
  { type: 'person.detected', camera: 'CAM-01' },
]

// Ready for backend connection:
const ws = new WebSocket('ws://localhost:3001/ws/events')
ws.onmessage = (event) => {
  const cloudEvent = JSON.parse(event.data)
  // Update UI with real-time data
}

Future API Endpoints:

GET /api/twin/buildings/{buildingId}/floors
GET /api/twin/floors/{floorId}/zones
GET /api/twin/floors/{floorId}/assets
GET /api/twin/zones/{zoneId}/telemetry
GET /api/twin/assets/{assetId}/status
WS  /ws/events (CloudEvents stream)

Part IV: What We Learned​

Making Autonomy Trustworthy​

The Three Pillars:

  1. Visibility - See what agents are thinking

    • Agent intent cards show reasoning
    • Decision trees displayed visually
    • State machine transitions visible

  2. Transparency - Understand why decisions are made

    • OPA policy explain shows exact rules
    • Audit trails with timestamps
    • Priority hierarchy clearly displayed

  3. Control - Human can intervene if needed

    • Override switches for emergency
    • Pause/resume agent actions
    • Manual mode fallback always available

Performance Surprises​

What Went Right:

  • ⚡ 60fps 3D rendering (smoother than expected)
  • 📦 Bundle size reasonable for prototype (1.73MB)
  • 🎨 three.js + React integration easier than feared
  • 🔄 State management straightforward (useState sufficient)

What Needed Optimization:

  • 📊 Too many re-renders initially (fixed with useMemo)
  • 🎭 Animation loops inefficient (moved to useFrame)
  • 🔍 Event bubbling caused issues (added stopPropagation)

UI/UX Discoveries​

Users Need:

  • 🎯 At-a-glance status - Red/yellow/green everywhere
  • 🔔 Alert prioritization - Don't overwhelm with noise
  • 📍 Spatial context - 3D helps understanding dramatically
  • 🕐 Historical view - Not just real-time, but trends

Developers Need:

  • 🛠️ Mock mode - UI development without backend
  • 📚 TypeScript types - Catch errors early
  • 🎨 Component isolation - Test each piece independently
  • 📊 Performance tools - React DevTools profiler essential

Part V: The Build Status​

Phase 2 Complete​

Dashboards Delivered:

  • ✅ Security Command Center (access control, surveillance, incidents)
  • ✅ Energy Operations Center (HVAC, cost, demand response)
  • ✅ Building Orchestrator (workflows, conflicts, coordination)
  • ✅ Gateway BFF (API aggregation, WebSocket streaming)

Components:

  • 📊 PolicyExplain - OPA decision visualization
  • 🔌 AgentDock - Multi-agent status panel
  • 🌐 MeshExplorer - Network topology view
  • 🎨 ConnectionStatus - System health indicator
  • 📍 GatewayOrigin - API endpoint display

Phase 3 Complete​

3D Digital Twin:

  • ✅ DigitalTwinSpatialView (full 3D building visualization)
  • ✅ ZoneOverlay (color-coded zones with telemetry)
  • ✅ AssetMarker (type-specific 3D geometries)
  • ✅ FloorSelector (multi-level navigation)
  • ✅ Interactive controls (toggle overlays, select objects)

Build Output:

dist/index.html                    0.58 kB │ gzip:   0.39 kB
dist/assets/index-CdbfiEYo.js  1,730.20 kB │ gzip: 508.05 kB
✓ built in 2.40s

Status: 🟢 All TypeScript errors resolved, build passing

Part VI: What's Next (Phase 4 Candidates)​

1. Asset Detail Modal​

When you click an asset, show:

  • 📊 Real-time telemetry charts (temperature, power, status history)
  • 📈 Historical data (24h/7d/30d trends)
  • 🎛️ Control actions (with OPA pre-check: "Can I unlock this door?")
  • 🔧 Maintenance history (last service, upcoming maintenance)
  • 🚨 Related incidents (security events involving this asset)

2. Time Travel & Replay​

The "killer feature":

  • ⏯️ Timeline scrubber - Rewind/play/forward controls
  • 🔖 Event bookmarks - Jump to important moments
  • 🎬 Incident replay - Watch how security event unfolded
  • 📊 Side-by-side compare - Before/after optimization
  • 🔮 Counterfactual - "What if we didn't unlock that door?"

3. Enhanced 3D Features​

Make it more realistic:

  • 🏗️ glTF models - Load actual building geometry (not just boxes)
  • 📐 BIM integration - Import Revit/AutoCAD models
  • 🎨 Texture mapping - Realistic surfaces (brick, glass, metal)
  • 🔍 LOD system - Level-of-detail for performance
  • 📦 KTX2 compression - Smaller texture sizes

4. Policy Studio​

Make policy editing safe:

  • 📊 Visual policy graph - See dependencies between policies
  • 🧪 Shadow mode - Test policy changes without affecting production
  • 📈 A/B testing - Compare policy versions on historical data
  • 🔄 Diff view - See exactly what changed
  • ↩️ Auto-rollback - Revert if policy causes problems

5. Multi-Building Portfolio​

Scale beyond single building:

  • 🗺️ Campus map - deck.gl geospatial overlay
  • 🌡️ Heatmaps - Temperature, occupancy, cost across campus
  • 🔗 Cross-building correlation - "Why are all buildings hot today?"
  • 🚨 Portfolio-wide incidents - Coordinate response across sites
  • 📊 Executive dashboard - KPIs for entire portfolio

Part VII: Alignment with LBI Spec​

Reference: Living Building Interface specification (see project documentation)

Phase 0 (Prototype) - COMPLETE​

✅ Load 3D floor, render zones/assets ✅ Pan/zoom/orbit at 60fps on mid-tier laptop ✅ Show agent intents (via AgentDock) ✅ Policy lens panel with sample OPA decisions ✅ No control actions yet (read-only and safe) ✅ Event stream integration (WebSocket with mock fallback) ✅ Floor selector for multi-level navigation

Phase 1 (Live Read-Only) - INFRASTRUCTURE READY​

✅ WebSocket to edge gateway (infrastructure complete) ✅ Live CloudEvents (mock stream working, ready for real) ✅ OPA explain read (PolicyExplain component active) ⏸️ Video tiles (planned for Phase 4) ⏸️ Replay timeline (planned for Phase 4)

Phase 2 (Supervised Control) - BACKEND NEEDED​

⏸️ Human-approved actions (UI ready, needs backend workflow) ⏸️ Undo/rollback (design complete, implementation pending) ⏸️ A/B policy testing (Policy Studio planned)

Note: See the full Living Building Interface specification in the architecture documentation and the technical milestone document in the project docs/milestones folder.

Part VIII: Technical Deep Dive​

Libraries & Versions​

3D Rendering:

  • three v0.160.0 - Core 3D engine
  • @react-three/fiber v8.15.14 - React renderer
  • @react-three/drei v9.103.0 - Helper components

React & TypeScript:

  • react v18.3.1
  • typescript v5.6.3
  • vite v5.4.11

Why These Choices:

  • three.js - Industry standard, huge ecosystem
  • R3F - React-friendly, declarative 3D
  • Vite - Fast builds, excellent DX
  • TypeScript - Type safety critical for complex state

Performance Optimizations​

Rendering:

// Use frustum culling (automatic)
// Shadow map resolution: 2048x2048 (high quality)
// Transparent materials for overlays (expensive but necessary)
// Mesh instancing for repeated geometries (future)

State Management:

// useState for local component state
// useMemo for expensive computations
// useFrame for animation loops (efficient)
// Event bubbling prevention (stopPropagation)

Bundle Optimization:

// Tree shaking (Vite default)
// Code splitting (route-based, future)
// Lazy loading (for 3D models, future)
// Asset compression (gzip: 508KB from 1.73MB)

Animation Patterns​

Zone Pulse Animation:

useFrame((state) => {
  if (isSelected) {
    const t = state.clock.getElapsedTime()
    meshRef.current.position.y = baseY + Math.sin(t * 2) * 0.2
  }
})

Asset Rotation:

useFrame(() => {
  if (isSelected) {
    meshRef.current.rotation.y += 0.01
  }
})

Status Ring Pulse:

useFrame((state) => {
  if (status === 'offline') {
    const t = state.clock.getElapsedTime()
    ringRef.current.scale.setScalar(1 + Math.sin(t * 3) * 0.2)
  }
})

Part IX: Code Quality & Testing​

TypeScript Coverage​

✅ All components fully typed ✅ Interface definitions for all data structures ✅ Props validation with TypeScript ✅ No any types used (strict mode)

Component Architecture​

✅ Separation of concerns (view/data/interaction) ✅ Reusable component design (ZoneOverlay, AssetMarker) ✅ Props-based composition (React best practices) ✅ React hooks for state management

Manual Testing Checklist​

  • Zone hover and selection
  • Asset hover and selection
  • Floor selector navigation
  • Toggle telemetry overlays
  • Toggle asset markers
  • Camera orbit controls
  • Performance on target hardware (60fps)
  • Multiple floor data load
  • Alert badge display
  • Status color accuracy

Future Automated Testing​

Unit Tests (Vitest):

describe('ZoneOverlay', () => {
  it('renders zone at correct position', () => {})
  it('changes color based on status', () => {})
  it('shows telemetry overlay when enabled', () => {})
})

E2E Tests (Playwright):

test('user can select zone and see details', async ({ page }) => {
  await page.goto('/')
  await page.click('[data-zone-id="lobby"]')
  await expect(page.locator('.zone-info')).toBeVisible()
})

Visual Regression (Chromatic):

  • Snapshot 3D views at different angles
  • Compare before/after component changes
  • Catch unintended visual regressions

Part X: The Bigger Picture​

Why This Matters​

Before the Living Building Interface:

  • Agents were black boxes
  • Policies were mysterious
  • Trust required blind faith
  • Debugging was impossible

After the Living Building Interface:

  • See agents thinking in real-time
  • Understand policy decisions with visual explain
  • Trust through transparency and visibility
  • Debug by replaying incidents in 3D

The Human-AI Collaboration​

This isn't about replacing humans. It's about amplifying human capability:

  1. Agents handle routine - 99% of decisions automated
  2. Humans handle exceptions - Complex conflicts escalated
  3. Agents learn from humans - Feedback improves future decisions
  4. Humans trust agents - Visibility builds confidence

Making Buildings Intelligent AND Trustworthy​

The CitadelMesh vision:

  • 🏰 Autonomous - Agents make decisions without constant supervision
  • 🛡️ Safe - OPA policies prevent harmful actions
  • 👁️ Visible - UI shows exactly what's happening
  • 🤝 Trustworthy - Transparency enables confidence

The Living Building Interface is how we achieve that trust.

Part XI: Developer Reflections​

What Surprised Us​

3D Web Tech Has Matured:

  • React Three Fiber makes 3D feel like normal React
  • Performance is excellent (60fps even with shadows)
  • Learning curve steep but worthwhile

Mock-First Development Works:

  • UI team unblocked from backend delays
  • API contract defined through mock data structure
  • Easy swap when backend ready

TypeScript Is Essential:

  • Caught dozens of bugs before runtime
  • Refactoring safe and confident
  • IDE autocomplete productivity boost

What We'd Do Differently​

Earlier 3D Prototyping:

  • Started with flat dashboards, then 3D
  • Should have prototyped 3D earlier to validate feasibility

More Design System:

  • Color palette defined ad-hoc
  • Should have created design tokens first
  • Would improve consistency

Performance Monitoring:

  • Added React DevTools profiler late
  • Should have monitored from day one
  • Caught some re-render issues late

Advice for Next Team​

  1. Start with mock data - Don't wait for backend
  2. Use TypeScript strictly - Enable all strict flags
  3. Profile early - React DevTools from the start
  4. 3D is feasible - Don't fear it, but respect the learning curve
  5. Component isolation - Build in Storybook or similar
  6. Accessibility matters - Color contrast, keyboard nav (we need to improve here)

Part XII: Conclusion​

What We Built​

Phase 2 (Command Centers):

  • 3 specialized dashboards (Security, Energy, Orchestrator)
  • Gateway BFF for backend aggregation
  • Real-time WebSocket streaming
  • Policy transparency components

Phase 3 (Digital Twin):

  • Full 3D building visualization
  • Interactive zone overlays with telemetry
  • Asset markers with status indicators
  • Multi-floor navigation
  • Performance optimized (60fps)

What We Proved​

✅ Autonomy can be visible - See agents thinking in real-time ✅ 3D web tech is production-ready - React Three Fiber is mature ✅ Trust comes from transparency - Show the "why" behind decisions ✅ Mock-first unblocks development - UI doesn't wait for backend ✅ TypeScript enables confidence - Refactor without fear

What's Next​

Immediate (Phase 4):

  • Asset detail modals
  • Time travel timeline
  • Video tile integration

Medium Term:

  • Policy Studio (visual policy editing)
  • BIM/glTF model loading
  • Enhanced telemetry charts

Long Term:

  • Multi-building portfolio view
  • Mobile app (React Native)
  • AR/VR building walkthrough

The Moment of Truth​

On October 13, 2025, we ran the build:

$ npm run build
✓ built in 2.40s

dist/index.html                    0.58 kB │ gzip:   0.39 kB
dist/assets/index-CdbfiEYo.js  1,730.20 kB │ gzip: 508.05 kB

No TypeScript errors. No warnings. Just a working 3D visualization of an intelligent building.

We opened the browser. The building appeared. Zones glowed with their status colors. Assets pulsed with their health. The floor selector showed all levels. The camera orbited smoothly.

For the first time, CitadelMesh was visible.

Not just logs and traces. Not just dashboards and numbers. But a living, breathing, thinking building that you could see and understand.

The agents were making decisions. The policies were protecting safety. The optimization was saving energy. And now, finally, humans could see it all working together.

This is how you make autonomy trustworthy. This is the Living Building Interface.

"We didn't just build a UI. We created a window into the mind of an intelligent building. Every color means something. Every animation has purpose. Every interaction builds trust. This is how humans and AI collaborate - with visibility, transparency, and beauty."

— CitadelMesh Chronicles, October 2025

Related Chapters:

Technical References:

  • Digital Twin Architecture - Full design specification
  • See project repository for technical milestone details and Gateway implementation

🏰 CitadelMesh - Making Autonomy Visible and Trustworthy

Chapter 13 Complete - October 13, 2025