NLP and Computer vision

What NLP and Computer Vision Are

Natural Language Processing (NLP) is the field of AI focused on enabling machines to understand and generate human language (spoken or written). Human language is unstructured: it contains ambiguity, idioms, variable word order, and contextual meaning that depends on world knowledge.

Computer Vision (CV) is the field of AI that enables machines to perceive, interpret, and generate information from images and videos. CV tasks include classification, detection, segmentation, and generation.

Both fields aim to convert unstructured sensory input (text, audio, pixels) into structured representations that downstream systems can act on.

---

Signals, segmentation, and structure in NLP

Machines break sentences into small fragments of information called signals.

Conceptual explanation

• Natural language must be preprocessed and segmented before higher-level reasoning. Segments can be tokens (words, subwords), syntactic phrases, semantic roles, or discourse units.
• Typical pipeline steps: text normalization - tokenization - morphological analysis - part-of-speech tagging - syntactic parsing - semantic parsing - pragmatic/discourse analysis.

Operational description

• Tokenization: splitting text into units. Modern systems often use subword tokenization (BPE, WordPiece, SentencePiece) to handle rare words and morphological variation.
• Syntactic parsing: produces a tree or dependency graph that describes grammatical relations.
• Semantic representations: can be shallow (named entities, key phrases) or deep (logical forms, frames).
• The signals that the Spanish text calls “signals” map to tokens, embeddings, spans, or features used by downstream models.

Technical comment

• The choice of segmentation affects model capacity and generalization. Subword tokenizers reduce the out‑of‑vocabulary (OOV) problem but introduce the need for careful detokenization. For languages with rich morphology, character-level or hybrid tokenization may be preferable.
• Modern transformer-based models take token embeddings (often positional) as input; these embeddings become the machine’s internal signals.

---

AI Debating

There are four core stages of an AI debating pipeline:

1. Learn and understand the topic
   • Retrieval of relevant documents (information retrieval).
   • Summarization and claim extraction: identify candidate claims, facts, statistics.
   • Topic modeling and framing: map retrieved evidence to themes.
2. Create a position
   • Decide a stance (pro/con) or multiple perspectives to present. This can be heuristic (maximizing evidence support) or learned (policy models trained on debates).
   • Balance between novelty, relevance, and defensibility of chosen points.
3. Organize the evidence
   • Rank and cluster supporting passages; build a coherent narrative.
   • Use rhetorical planning (what goes first, what supports which claim).
   • Prepare citations and indicate confidence or provenance for claims.
4. Respond to opponents
   • Detect opponent claims and align them with the system’s evidence graph.
   • Generate rebuttals: contradiction, weakening via counter-evidence, or reframing.
   • Maintain coherence and avoid logical fallacies.

This pipeline mixes classical IR (BM25, dense retrieval using vector embeddings) with modern NLU: neural rankers, claim detection classifiers, and controlled text generation. Evaluation uses metrics from IR (precision/recall), argument quality scoring (human evaluation), and factual consistency tests.

---

Emotion detection (Affective computing)

Emotion detection labels different kinds of human emotion from data (text, audio, images).

Conceptual explanation

• Emotion detection models map input (e.g., text sentence, voice recording, facial expression) to emotion labels such as joy, sadness, anger, surprise, fear, disgust, or more granular affective states.

Operational description

• Text: supervised classifiers trained on labeled corpora (tweets, movie reviews, annotated dialogs) or weak supervision methods.
• Audio: features like pitch, energy, spectral features feed into sequence models (RNNs, CNNs, Transformers).
• Vision: facial action units or CNN-based classifiers on face crops.

Emotions are culturally and contextually dependent. Models must be validated across demographics to avoid biased inferences. Multi-modal approaches (text+audio+video) tend to be more robust because they integrate multiple cues.

---

Sentiment analysis vs. Emotion detection

Sentiment analysis measures intensity and polarity (positive/negative/neutral), while emotion detection identifies specific emotions.

Explanation

• Sentiment analysis (opinion mining) is usually coarser: polarity and sometimes intensity (e.g., 1–5 stars) and aspect-based sentiment (opinions about specific attributes).
• Emotion classification assigns categorical emotional states or continuous affective dimensions (valence, arousal, dominance).

Sentiment models often use binary or ternary classifiers; more advanced models use regression to predict continuous scores. Evaluation metrics include accuracy, F1, MAE (for regression), and correlation with human ratings.

---

Chatbots: Frontend and Backend

Chatbots interact with users (frontend) and run logic/memory on the backend.

Frontend (user interaction layer)

• Presents the user interface (chat window, voice UI) and handles I/O: speech-to-text, text display, text-to-speech.
• Should guide the user with prompts, clarifying questions, and feedback.

Backend (logic and memory)

• Dialogue manager: controls conversational flow and state.
• NLP modules: intent recognition, entity extraction, context tracking, response selection/generation.
• Memory: short-term session memory (dialogue turns) and optionally long-term memory (user preferences).
• Integration: connects to databases, APIs, knowledge bases, and external services.

Designing memory schemas is critical. A simple finite-state or slot-filling bot suffices for transactional flows (bookings, FAQs). For open-domain or mixed-initiative dialogue, use transformer-based dialogue models, retrieval-augmented generation (RAG), and explicit context windows.

For voice-enabled chatbots, pipeline: microphone - VAD (voice activity detection) - ASR (automatic speech recognition) - NLU - DM - NLG - TTS.

---

Intents, Entities, and Dialog Flow

A chatbot identifies intent (what the user wants), entities (nouns/objects), and uses these to trigger a dialog.

Definitions and examples

• Intent: the purpose behind a user utterance. Example intents: book_flight, check_balance, cancel_reservation. Think of intents as verbs/actions.
• Entity: a value or object associated with the intent. Example entities: destination, date, amount. Think of entities as nouns.
• Dialog: the scripted or dynamic flow of conversation that satisfies an intent by collecting required entities and performing actions.

Dialog design approaches

• Rule-based / flowchart: explicit dialog trees (good for narrow tasks and safety-critical systems).
• Statistical / learned: policies learned with reinforcement learning or supervised learning; these generalize but need more data.
• Hybrid: use rule-based safety layers + learned NLG for naturalness.

Intent classifiers are commonly fine-tuned transformer encoders (e.g., BERT). Entity extraction uses sequence labeling (BiLSTM-CRF historically, now token classification heads on transformers). Dialog managers can be implemented with POMDPs or RL agents when optimizing for long-term goals.

---

Computer vision: CNNs and GANs

Convolutional neural networks (CNNs) analyze images; generative adversarial networks (GANs) create images.

Convolutional Neural Networks (CNNs)

• Purpose: learn hierarchical features from images: edges - textures - object parts - objects.
• Architecture elements: convolutional layers, pooling, batch normalization, activations (ReLU), and fully connected layers or global pooling.
• Use cases: image classification, object detection (with region proposal or single-shot detectors), semantic/instance segmentation.

Technical comment about CNNs

• Modern CV often uses residual connections (ResNet), dense connections (DenseNet), or attention modules (SE blocks). For many tasks, transformer-based Vision Transformers (ViT) are now competitive and sometimes preferred for large-scale training.

Generative Adversarial Networks (GANs)

• Purpose: learn to generate realistic images by pitting two networks against each other: a generator (G) and a discriminator (D).
• Training idea: G tries to create images that fool D; D learns to distinguish real from fake. The min-max game yields realistic samples when balanced correctly.