Forward Deployed Engineer (FDE), Language Models
ARTIFICIAL ANALYSIS
We're an independent AI benchmarking company that benchmarks language models, inference providers, hardware, image/video and speech and publishes a widely cited Intelligence Index. This role runs the LLM benchmarking stack and works directly with AI labs; the company is also hiring multiple MTS roles across evaluations, inference, hardware and speech and other engineering and product roles.
What you’ll do
- Run LLM benchmarking stack and work directly with AI labs (FDE)
- Build frontier evals including datasets, harnesses and benchmarks (MTS, Language Model Evaluations)
- Benchmark speed, quality and price across serverless inference providers (MTS, Inference)
- Benchmark GPUs, TPUs and custom silicon (MTS, Hardware)
- Own TTS, STT and speech-to-speech evals (MTS, Speech)
What they’re looking for
- Experience building evals, datasets, harnesses or benchmarks
- Ability to work directly with AI labs on pre-release models
- Experience benchmarking inference and hardware or working on speech/ML systems
Skills
- large language models: LLM, language models, language model evaluations, LLM benchmarking
- inference providers: inference providers, serverless inference, benchmark speed quality price
- hardware: GPUs, TPUs, custom silicon, benchmark GPUs TPUs
- speech: TTS, STT, speech-to-speech, speech evals
- evaluations & benchmarking: datasets, harnesses, benchmarks, Intelligence Index
The post
ARTIFICIAL ANALYSIS | MTS, Forward Deployed Engineer + more | San Francisco, CA (ON-SITE, 5 days) or Melbourne, Australia (ON-SITE, 3 days) | Full-time | https://artificialanalysis.ai/careers
We're an independent AI benchmarking company. We benchmark language models, inference providers, hardware, image/video and speech, and our Intelligence Index is widely cited when new models launch. Team of ~50, backed by Nat Friedman, Daniel Gross, Andrew Ng and others.
The people who build our evals also work directly with the AI labs, often on pre-release models. Flat structure, lots of ownership.
Main hiring needs:
* Forward Deployed Engineer (FDE), Language Models: run our LLM benchmarking stack and work directly with the labs
* Member of Technical Staff (MTS), Language Model Evaluations: build frontier evals (datasets, harnesses, benchmarks)
* MTS, Inference: benchmark speed, quality and price across serverless inference providers
* MTS, Hardware: benchmark GPUs, TPUs and custom silicon
* MTS, Speech: own our TTS, STT and speech-to-speech evals
Also hiring for media generation, full stack, ML engineering, robotics and product roles.
Apply at https://artificialanalysis.ai/careers.