How to Become an LLM Engineer: A Knowledge Map and Learning Method

type

status

date

slug

summary

Who am I?

I’m Rui, a research scientist at Meta working on LLMs and large-scale recommendation systems. One recent project gives users direct control over their feeds through natural language: what they want to see more of, what they want less of, and how long the change should last.

That problem sits between search, recommendation, and conversational AI. A request must become a precise instruction. Relevant content must be found and ranked. The system must apply the change, respect its scope, and let the user reverse it.

Previously, I worked on ad targeting and recommendation at Amazon. I completed my PhD at the University of Maryland, studying catastrophic-event forecasting with deep learning. I care about AI safety and alignment, particularly how to give increasingly capable systems useful but bounded authority.

Why should you read this series?

I spent two and a half months preparing for a research engineering role at OpenAI. I am now preparing to join its Agent Integrity Research team.

This series turns the knowledge map behind that preparation into technical explanations, worked examples, and implementations. It also includes areas I intend to study more deeply, especially post-training and inference. It is not a claim that I mastered every topic during those two and a half months. I started with a PhD and years of applied ML experience.

The intended reader can write Python and knows basic machine learning, but wants to connect that background to LLM work. The question is not just “What should I read?” It is “What should I be able to explain, implement, and diagnose after reading it?”

I will cover the material in seven areas. Each area below includes its scope and a concrete example of the understanding it requires. This is my study map, not an official hiring rubric or a guarantee of a particular job.

The LLM engineer knowledge map

In this series, LLM engineering includes both building systems around models and understanding how models learn. You do not need equal depth everywhere. But you need enough breadth to recognize whether a failure comes from the data, the model, retrieval, execution, or the measurement itself.

0. Foundations: turn an idea into correct math and code

The foundation is probability, conditional distributions, expectation, gradients, and optimization. The implementation layer is PyTorch: tensor shapes, broadcasting, indexing, masking, and vectorization. The basic workflow is data cleaning, a baseline, training, evaluation, and error analysis. Debugging means inspecting loss curves, overfitting, gradients, and numerical stability.

Here is a small example you can run:

The predictions match the targets. Yet the first loss is not zero. Broadcasting produces a (3, 3) matrix, comparing every prediction with every target instead of comparing matching pairs. The code runs; the objective is wrong. PyTorch’s broadcasting rules explain why.

A useful test: implement a masked loss with a loop, vectorize it, and compare both values and gradients. Test unequal sequence lengths and decide what an all-masked batch should do. Tensor Puzzles provides focused practice in this style.

This is why I treat tensor fluency as part of understanding an algorithm, not a separate coding exercise.

1. Data and evaluation: decide what evidence to trust

This area starts before the model. It includes annotation instructions, disagreement, confusion matrices, noise, and bias. I will cover majority vote and weighted aggregation, then label-inference methods such as Dawid–Skene and MACE. These methods deserve attention because annotators need not be equally reliable; MACE explicitly models annotator competence while recovering labels.

The next layer is experimental validity: data splits, leakage, trusted reference examples, baselines, ablations, and error analysis. Evaluation topics include precision–recall metrics, Recall@K, NDCG, ROUGE, and rare-event testing. For agents, the map extends to human-calibrated LLM judges, component tests, end-to-end outcomes, and safety requirements. Online work adds randomization, A/B tests, uplift, and guardrail metrics.

Consider a hypothetical safety detector screening 10,000 requests. Suppose 100 are unsafe. At a threshold with 90% recall and a 1% false-positive rate, it flags 90 unsafe requests and 99 safe ones. Only about 47.6% of its alerts are true positives. Reporting recall alone hides the review burden. A detector that labels everything safe would have 99% accuracy and catch nothing.

Even metric names need care. Average precision and trapezoidal area under a precision–recall curve are not the same calculation. Scikit-learn documents the distinction.

A useful test: write a grading rubric, have humans and an LLM grade the same examples, and inspect disagreements by failure type. For actions, check the resulting state, not just whether the response sounds successful. Anthropic’s agent-evaluation guide makes this transcript-versus-outcome distinction explicit.

My priority here is to make the training signal and the evaluation target inspectable. Otherwise, a better score may mean we became better at satisfying a flawed measurement.

2. Models and learning: understand what changes the model’s behavior

The starting topics are loss functions, optimization, generalization, and training stability. Representation learning adds embeddings, contrastive learning, SimCSE, and CLIP. Language modeling adds tokenization, transformers, and generation. Diffusion belongs to the broader generative-model branch, but is not a prerequisite for this LLM path.

For decision learning, I will start with contextual bandits: exploration, exploitation, UCB, and Thompson sampling. The deeper RL path connects reward and return to value functions, Bellman equations, temporal-difference learning, SARSA, and Q-learning. It then moves to policy gradients, REINFORCE, actor–critic methods, and generalized advantage estimation.

For LLM post-training, the first distinction is the learning signal. Supervised fine-tuning fits demonstrations. Preference methods use comparisons between outputs. Policy-optimization methods learn from rewards on sampled behavior. The InstructGPT paper connects demonstrations, a learned reward model, and RL. DPO offers a preference-learning route without training a separate explicit reward model.

A worked comparison can start with one prompt, a preferred answer, a rejected answer, and several scored samples. What information does each training objective use? Which probabilities does it change? What feedback is missing? We will derive those differences before comparing algorithm names.

The advanced branch includes importance sampling, KL regularization, clipping, and token-level credit assignment. It leads into PPO, GRPO, DAPO, GSPO, and CISPO. PPO and DeepSeekMath, which introduced GRPO, are starting references, not interchangeable recipes.

A useful test: take one update and identify the sampled data, the objective, the trainable parameters, and the quantities treated as fixed. Explain why each term is present before implementing it. This is the level at which similar-looking formulas stop being interchangeable.

3. Search and recommendation: find the right candidates before ranking them

The map starts with intent and relevance: what the user explicitly requested versus what behavior suggests they might prefer. Retrieval topics include keyword matching, learned sparse representations such as SPLADE, dense embeddings, nearest-neighbor search, and approximate indexes. Ranking topics include candidate generation, reranking, hybrid retrieval, deduplication, score fusion, and LLM rerankers.

One useful architectural distinction is where query–document interaction happens. A single-vector retriever compresses each side before matching. ColBERT retains token-level representations and combines them at scoring time. Documents can still be encoded offline. That is the point of its late-interaction architecture, not merely a different similarity function.

Now consider a fixed candidate set. A collection has ten relevant documents for a query, but the retriever returns only six of them. A reranker that only reorders those candidates cannot recover the other four. Its maximum recall over that set is 60%, even with perfect ordering. If the goal is higher total recall, the next intervention should address candidate coverage, not automatically add a larger reranker.

Recommendation adds user representations, semantic tags, frequency controls, diversity, and explicit preference changes. “Show me less of this” is a control request, not just another behavioral signal to mix into a score.

A useful test: compare retrieval systems on the same labeled queries. Separate missing candidates from badly ordered candidates. Then compare the marginal quality gain, latency, and cost of reranking. The diagnosis should determine which stage you change.

4. Agents: translate language into bounded actions

The agent branch covers structured intent, plans, retrieval-augmented generation, context selection, tool execution, state, and memory. Reliability adds admission checks, permissions, validation, reading back saved state, expiry, editing, undo, fallback, and rollback.

For example, take the request “Show me less basketball for two weeks.” A structured interpretation needs at least a topic, a direction of change, a scope, and an expiry. The executor must check whether the action is supported and authorized. After saving it, the system should verify what actually changed. A later “Undo that” needs to identify the relevant change, not merely produce an apology. This illustrates the kind of feed-control problem behind my work; it is not a description of an internal implementation.

A second distinction is between a fixed workflow and an agent that chooses its next step. Anthropic’s architecture guide separates predefined code paths from model-directed tool use. That distinction gives us a concrete design question: where does the task require adaptive decisions, and where should ordinary code remain in control?

RAG supplies retrieved information to generation; it does not by itself define an action policy. The original RAG paper is a reference for the retrieval–generation connection, not a specification for permissions or memory updates.

A useful test: give a system an ambiguous request, an expired preference, an unauthorized action, and a failed tool call. Specify acceptable behavior for each. Evaluate both the response and the stored state.

5. Production systems: account for time, memory, and partial failure

The systems branch includes data modeling, caching, sharding, consistency, incremental synchronization, and permissions. ML serving adds offline versus online computation, latency, throughput, and separating slow reasoning from fast execution. LLM inference adds key–value caching, batching, quantization, and distillation. GPU programming and inference kernels are deeper extensions.

Two examples show why these are not merely deployment details.

First, a calendar service saves an event, but the response is lost. A retry can create a duplicate unless the API recognizes the same operation. A timeout means the caller does not know what happened; it does not prove that nothing happened. AWS’s discussion of idempotent APIs explains how request identity can make retries safe. The same issue appears when an agent invokes a tool twice.

Second, model weights are only part of inference memory. For a conventional full-attention transformer cache, an illustrative storage calculation is:

For one sequence with 32 layers, 32,768 cached tokens, 8 KV heads, a head dimension of 128, and two-byte values, the cache alone is 4 GiB. This excludes weights, activations, and allocation overhead. It assumes no cache compression, sliding window, or sharing. The PagedAttention paper explains how cache allocation and fragmentation constrain serving batch sizes.

A useful test: estimate memory before running a workload, then compare the estimate with measurements. Measure time to first token, generation speed, throughput, and tail latency separately. Inject timeouts and repeated requests into state-changing tools. Check whether retries preserve the intended outcome.

6. Research judgment and technical leadership: choose and defend the work

This area covers problem selection, user value, constraints, success criteria, alternative designs, experimental evidence, failure reviews, and attribution. It also includes shared evaluation standards, ownership boundaries, and release decisions across teams.

Suppose a system improves after changing the retriever, prompt, and model together. That may be a useful product result, but it does not isolate which change caused the improvement. To investigate, compare controlled variants and state what each comparison can establish. Interactions may require more than changing one component at a time.

I want to practice writing down the reasoning behind a technical decision: the baseline, the observed failure, the alternatives, the evidence for the chosen intervention, its cost, and the remaining uncertainty. For a team project, I also need to distinguish the team’s outcome from the decisions and implementation I personally owned.

A useful test: write a one-page decision record that another engineer can challenge. Include what evidence would make you reverse the decision. A design that cannot be questioned precisely is difficult to improve or hand over.

How the areas connect

The map is not seven independent courses. A mislabeled example can distort both training and evaluation. Missing retrieval candidates can look like a generation problem. A tool retry can turn a correct interpretation into an incorrect state change. A larger context can improve available evidence while increasing memory use.

This suggests a way to choose what to study next: identify the failure, locate the layer responsible, and deepen the knowledge needed to test a repair. Do not reach for post-training when the relevant evidence was never retrieved. Do not spend the evaluation budget on a larger judge before clarifying the grading rule.

How should you prepare?

Set a target and test your gaps

“Learn LLMs” is too broad to organize a week of work. Choose a capability: implement a training objective, diagnose a retrieval pipeline, or build a reliable tool-using system. Use the map to identify its dependencies.

For each topic, distinguish three levels: I recognize the terminology; I can explain the mechanism without notes; I can implement it and handle a changed example. A reading plan should reflect the difference. Recognizing “importance sampling” is not evidence that you can derive or debug the weighting in an update.

For an application-focused path, prioritize foundations, evaluation, retrieval, agents, and systems. For a post-training path, put more depth into probability, optimization, data quality, and learning objectives. For inference work, emphasize tensor operations, architecture, memory, and serving measurements. These are suggested starting paths, not separate professions with fixed borders.

Use AI to diagnose understanding

My preparation included using ChatGPT and Codex as learning tools. The important part of the method is not asking for more explanations. It is testing whether I can produce one and defend it.

A study instruction I recommend is:

Ask me to explain the mechanism before teaching it. Identify the most important error. Give one counterexample and let me revise my answer. Then change an assumption and test whether I can still reason through the problem.

Pair that interaction with the paper, documentation, and executable checks. For example, after explaining broadcasting, predict the output shape before running the code. After explaining a metric, calculate it on a tiny dataset. An assistant’s agreement is not independent evidence that the explanation is correct.

Protect focus, but define a stopping condition

Set aside uninterrupted time for one dependency chain. A session on policy gradients may need to return to expectation and the log-derivative identity. That is relevant work. Opening unrelated model announcements is not.

I recommend giving each session a concrete finish line: derive one identity, reproduce one result, or fix one bug with a regression test. This makes “study longer” a deliberate choice rather than a substitute for knowing what remains unclear.

Determination matters when progress is slow. But confidence should come from accumulating evidence: an explanation that survives a counterexample, code that passes an edge case, or an experiment that resolves uncertainty. It should not require pretending every current belief is correct.

Turn the lesson into an artifact

For an implementation post, the useful artifact is code with inputs, setup instructions, and checks. For a conceptual post, it may be a derivation or a worked comparison. For an experiment, it is the question, baseline, configuration, result, and limitations.

Then explain what the artifact taught you. Did a familiar method fail under a different assumption? Was the bottleneck in a different layer than expected? Did the result rule out an attractive idea? Those are more useful contributions than a list of tools used.

This is also why I am writing publicly. I want to improve my ability to explain technical ideas, build usable solutions, and share conclusions that other people can inspect and challenge.

The learning skill underneath the roadmap

The learning process I want to develop is: set a goal, build a map, diagnose gaps, practice the missing pieces, and teach the result. A separate set of posts will examine how to learn a new domain, retain the important ideas, and communicate them clearly.

My standard for understanding is stricter than recognizing an explanation. I should be able to restate the mechanism, distinguish it from a similar idea, reconstruct a small version, identify where it fails, and transfer it to a different problem. The test is especially valuable at the step I am tempted to skip.

For the broadcasting example, that step is naming the axes and predicting which elements are paired. For retrieval, it is separating candidate coverage from ordering. For agent actions, it is separating a statement of success from a verified state change. Those distinctions are the kind of material I want each post to make concrete.

What comes next

The engineering series will expand individual branches of this map through focused explanations and working examples. The overview is a navigation page, not a promise to cover every topic before building anything. Deeper RL and GPU topics will be marked as extensions.

A later LLM researcher series will focus on question selection, reproducing work, designing controlled experiments, interpreting negative results, and deciding what a result actually establishes. The distinction I will use is practical: engineering asks how to make a system work under constraints; research asks what we can learn that was not established before. Both need careful implementations and credible measurements.

To begin, choose one diagnostic from the map and attempt it without an assistant writing the answer first. Record the exact point where you get stuck. That gives you a specific next lesson rather than another open-ended reading list.

The views are my own. Technical examples use public sources or explicitly stated assumptions, not confidential employer systems.