Skip to content
← Back

Case study

Award-travel discovery & feasibility engine

LLM discovery + deterministic scoring for a Gulf airline loyalty programme

Sanitized production case studyRole · Designed and evaluated end-to-end

A multi-stage destination-discovery and award-feasibility engine for a Gulf airline's loyalty programme — turning a member's intent into ranked, explainable destination and routing suggestions, benchmarked against a baseline before shipping.

Problem

Members wanted to know where their loyalty currency could actually take them — not just whether one route was bookable. The existing rule-based logic was rigid and opaque: it couldn't reason over fuzzy intent (somewhere warm in spring, direct, under a given budget) or explain why a suggestion was made.

My responsibility

Designed and evaluated the discovery pipeline end-to-end: intent parsing, LLM-driven destination discovery, deterministic backend scoring, and an explainability layer. Ran head-to-head benchmarks against the existing approach and proposed the API changes that made it production-viable.

Key decisions

  • Split the pipeline into four explicit stages — intent parsing, LLM discovery, deterministic scoring, explainability — so the creative step and the trustworthy step are separated and independently testable.
  • Keep ranking deterministic: the LLM proposes candidates, but feasibility and ordering are computed by backend rules over real fare and availability data.
  • Benchmark a local pipeline against a RAG-hybrid variant before committing, rather than assuming the heavier approach wins.
  • Collapse many per-destination availability calls into a single batched request to make the engine viable under real latency and rate limits.

Architecture

Win
outperformed the baseline in head-to-head evaluation
↓ lat
measurable median-latency improvement
Batched
many per-destination availability calls collapsed into one request
Multi
distinct routing output types
  • Deterministic / validated
  • LLM / generative (bounded)
  • Orchestration / boundary
  • Neutral component
Discovery pipelineB1 · LLM proposes, scoring decides
Memberquery01Intent parsing02 · LLMDiscoveryproposes candidates03 · RULESScoringdeterministic rank04ExplainabilityRankedresultsFaresAvailabilitymany calls → one batched request
The creative step and the trustworthy step are separated. The LLM expands intent into candidate destinations; deterministic rules score and rank them against real fare and availability data, so results stay reproducible and defensible.
Benchmark: RAG-hybrid vs baselineB2 · internal eval
Head-to-head win ratelowerBaselinehigherRAG-hybridMedian latency (relative)baselineBaselinelowerRAG-hybridmeasurable improvement
Measured on a fixed query set before committing to the heavier approach. Bars are qualitative: the RAG-hybrid variant outperformed the baseline on head-to-head quality and on median latency. Exact figures are sanitized and available under NDA.

Stack

  • Python
  • Azure OpenAI
  • Hybrid RAG
  • Batched availability API
  • Evaluation harness

Outcome

  • · A RAG-hybrid pipeline that outperformed the rule-based baseline in head-to-head evaluation, with a measurable latency improvement.
  • · An award-feasibility routing engine producing multiple distinct output types (direct, one-stop, near-date, points-stretch).
  • · Specification for a batched availability API collapsing many per-destination calls into a single batched request.

Known limitations

  • · Benchmark results come from internal evaluation on a fixed query set, not a public leaderboard.
  • · LLM discovery can propose destinations that scoring later rejects — quality depends on keeping the scoring layer authoritative.
  • · Results are only as fresh as the underlying fare and availability feed.
  • · Client details and specific figures are sanitized and available under NDA during a technical interview.

Sanitized details available during technical interview.