Friendularity.org

Character Architecture - Overview Presentation

Target running time:  20 minutes

I) Overview

We will cover the following main topics:

1. Character Interaction : Auditory+Verbal and Visual+Animation+Embodiment -   10 minutes 
2. Character Cognition - 5 minutes 
3. Character Platforms and Technology Integration - 5 minutes 

II) Character Interaction : Auditory+Verbal and Visual+Animation+Embodiment-in-Environment

II.A) Character Development (/Authoring) vs. End-user Interactions

Ideally and eventually this is a seamless spectrum of activity, but today, on the ground, we must make many compromises.

II.B) Auditory and Verbal Interaction

1. Characters hear through speech recognition, audio source localization (connecting us to Embodiment below), musical beat detection, sample recording for mimicry, and other techniques.
2. Characters speak, sing, and make emotive sounds using text-to-speech, text-to-song, parametric sound synthesis, and many techniques available in electronic music space, including synchronization + triggering to/from external systems via MIDI, VST, SMPTE, etc.

II.C) Character Animation and Embodiment-in-Environment

1. Animation occurs through tunable blending of parametric gestures at varying granularity.
   1. Gesture blending, prioritization, and sequencing are key challenges.
   2. Each feature implemented by replacable and connectable modules, with limited technical and semantic coupling.
2. Animation is rendered through bones of an Open GL 3D onscreen model, and/or an embodied robot's servo channels.
   1. Physical robot animation done by embedded computer or over serial channels.
   2. Onscreen 3D character models are defined and exchanged using popular formats like Collada, X3D, Ogre, Blender, Maya.
3. Character expression and reaction to stimuli is driven through both dynamic cognition and triggering of authored content.
   1. Authoring and testing is done at varying granularity, from a single curve to a branching orchestral sequence.  
4. Synchronization with audio/speech output is baked in to the technology platforms (e.g. via MIDI triggering of animation).
5. Character is aware of physical position, orientation, momentum and surroundings in some Environment, via sensors and/or virtual world information.
   1. Auditory (e.g. in Audio Localization above) and Visual information contributes to Understanding of Environment.
   2. Character maps and navigates using SLAM when appropriate.
   3. Tactile and obstacle feedback information may be sensed or virtually mapped in.
   4. Kinematics and dynamics including gravity may be real or simulated.
   5. Character's line of sight is a principal vector for analysis, execution, and authoring.

II.D) Character Vision and Response to Visual Stimuli

1. Face and Motion Saliency Tracking
2. Facial Expression Analysis
3. Gesture Analysis
4. Integration of Character Response with Expressive Behavior

III) Character Cognition

• PUMA = Perceive, Understand, Motivate, Act
  • Builds upon BDI = Beliefs, Desires, Intentions
  • Cognition can be reactive, from perception to action.  If U+M are skipped or trivial, we have a reflexive behavior.
  • Cognition can also have internal dynamics in the U+M stages.
• Cognition is carried out in distributed fashion across local hosts (embedded + desktop/laptop) and the network cloud of services.
  • Meaning and names are resolved using formal ontologies and RDF semantic technology.  
    • URIs and RDF are our common cognitive currency.
  • Symbolic, Embodied, Adaptive, Neural, Hormonal, and Genetic thinking, learning and memory is bridged in through many available open tools and formats 
    • Mixed-symbolic approach of ACT-R combines well with symbolic ruminators like Cogbot, LARKc (including Web PIE for Hadoop), Open Cyc.
    • Endocrine simulation (hormones) is an active area of work.
    • Offline/online data mining using Weka and Pentaho as well as Hadoop stack including Mahout.
    • Robotic mind federation via bridges to Darwin, YARP, ROS.
• Cognitive processes are simulated, studied, and developed offline in many signal + agent laboratory environments (Sci Lab, R, Protege), some of which have deployable forms.

IV) Character Platforms and Technology Integration

IV.A) Platform Core Capacities

1. Core platform components are Java OSGi modules.  We lean towards Apache Felix stack for deployment, but this choice is variable by target and goals.
2. Native code (C++) features are java-wrapped and OSGi-bundled for platform agnosticity.
3. Target platforms are Linux, Win, Mac including Android and Java ME profiles.
4. Cognitive resolution core is grounded in RDF semantic technology.
   1. Vocabularies defined using ontologies, usually OWL-DL.
   2. SPARQL supported as both client and server.
5. Higher order computing is done within JVM using: 
   1. Functional paradigm: Scala
   2. Description logic and frame paradigms: OWL 2.0, DL-Safe Rules, Jena Rules
   3. Backward chaining logic: Prova
   4. RETE forward chaining: Drools or Jena Reasoner
   5.  Any other JVM-friendly language (Clojure, Jython, JRuby, Groovy, etc.)
6. Mathematics are algebraically grounded when practical, using JScience.org's  Group, Ring, Field, Polynomial, Rational Function.   Runtime symbolic calculus is available.
7. Open GL rendering done via J Monkey Engine.

 IV.B) Platform Interoperability and Integration

1. Interop and Integration via many pathways:
   1. Any java-specific path like OSGi, RMI, JMX, JBI, JMS, EJB
   2. SQL interaction via JDBC / ODBC connection to H2 (which can run inside our Java process), My SQL, Oracle, etc.
      1. RDF may be stored+queried in SQL DBs.
   3. Information sharable at semantic level as RDF, governed by formal ontologies, usually OWL-DL.
      1. Implementation leans heavily on Jena and in some cases Pellet.
      2. Serialization to XML, Turtle, JSON, etc.
      3. Native code (ugh!) may interoperate at semantic level using Redland.
   4. Object sharing via Avro(multi-platform, optional-schema, JSON-friendly) serialization and the following pathways:
      1. Platform-agnostic AMQPpub/sub protocol implemented by 
         1. Apache QPid (Java/C++ broker, plus client bindings for .NET, etc) 
         2. Rabbit MQ (Erlang broker, again many language bindings).
      2. Shared blackboard network hashtable caches implemented by Memcached.
      3. Hadoop services for large scale computing
   5. MIDI and VST(virtual synthesizer) integration for synchronized timelines - connectivity to huge number of well supported and documented, free or COTS programs and devices such as Ableton, Psycle, Frinika, Cubase, Logic, Pro Tools...
   6. Numeric and media dataset sharing via HDF5.
   7. Robotic federation via bridges to Darwin, YARP, ROS.
   8. Semantic federation via bridges to Cogbot, LARKc, jACT-R.

IV.C) Infrastructure

1. Java and Scala modules are built and tested using Netbeans, Eclipse, Maven, and Ant.
2. Preferred OSGi container is Apache Felix and derivatives.
3. Java logging is routed through SLF4J.
4. Amazon S3 services is preferred cloud computing platform.
5. Assembla.com services for code hosting and project communications.