Back to all projects
Compilers & NLP Engineering20244 Weeks Engineering
VarToken Studio
Manual automata-based tokenizer for noisy, code-mixed Roman Urdu text with DFA engine and Turing visualizer.
Client EntityCS Core / Theory of Automata Research
My RoleSystems & Algorithm Engineer
Delivery Time4 Weeks Engineering
Tech StackTypeScript, DFA / NFA State Machines + 3 more
01 // The Problem
Friction, Inefficiency & System Barriers
Standard NLP tokenizers fail catastrophically on Roman Urdu (e.g., "kyunnn", "kese ho", "btao") because phonetic spelling is informal, lacks standard lexicons, and varies wildly across users.
02 // My Approach & Decisions
Architectural & UX Decisions
Built a Deterministic Finite Automaton (DFA) state transition pipeline with character stream normalization, repeated-letter folding, and an interactive 60 FPS visual Turing tape debugger.
Application Previews & Workflow
03 // Technical Depth
Code Moment: Engineered Core
DFA State Transition & Tape Head Step
typescript
| 1 | export function stepDFA(currentState: DFAState, char: string, tapeHead: number): DFATransitionResult { |
| 2 | const normalizedChar = normalizePhoneticChar(char); |
| 3 | const nextState = TRANSITION_TABLE[currentState]?.[normalizedChar] ?? DFAState.DEAD; |
| 4 | |
| 5 | return { |
| 6 | previousState: currentState, |
| 7 | currentState: nextState, |
| 8 | consumedChar: char, |
| 9 | tapePosition: tapeHead + 1, |
| 10 | isAccepting: ACCEPTING_STATES.has(nextState) |
| 11 | }; |
| 12 | } |
Executes formal automata transitions over the input stream, advancing the virtual tape head and classifying valid tokens.
04 // Measurable Results
Verified Production Benchmarks
10K tok/sParsing Throughput
60 FPSRender Frame Rate
100%Grammar Coverage
92%Noise Reduction
Key Capabilities Delivered
- Real-time character-by-character DFA state transition debugger
- Turing tape visualizer tracking tape heads and cell states
- Noise normalization pipeline (slang reduction, repeated letter filtering)
- Contextual semantic meaning explanation layer
Full Technology Ecosystem
TypeScriptDFA / NFA State MachinesTuring Tape VisualizerCanvas APIGemini API