Staff Software Engineer - ML Observability - Datadog

Fully remote

General
Added
Type
Full-time
Salary
$234k - $300k

DESCRIPTION FORMATTING RULES (CRITICAL - FOLLOW EXACTLY)

Your job is to output clean, readable Markdown while PRESERVING ALL ORIGINAL CONTENT AND STRUCTURE EXACTLY.

ABSOLUTE RULES - NEVER VIOLATE:

  1. NEVER add, remove, or modify any text/wording from the original
  2. NEVER paraphrase or rewrite sentences - keep exact wording
  3. NEVER remove sections, paragraphs, disclaimers, or any content
  4. PRESERVE all links exactly as they appear
  5. PRESERVE the natural flow and spacing of the original document

MARKDOWN FORMATTING:

  • for main section headings (bold standalone titles like "The Role", "You Will", "Requirements")

  • for sub-section headings

  • NEVER use # (H1) - the job title is already the page H1
  • Use "-" bullet lists for ANY list of items
  • CRITICAL: If content is clearly a list (requirements, responsibilities, benefits), format as bullet list
  • Bold for inline emphasis, key terms, skills - preserve existing bold from original
  • Italic for emphasis or disclaimers - preserve existing italic from original
  • text for ALL hyperlinks - preserve every link

SPACING RULES:

  • Preserve the natural paragraph breaks from the original content when they exist
  • IF the original lacks proper spacing (e.g., everything is one giant paragraph), ADD logical breaks:
    • Blank line before each section heading
    • Blank line between distinct topics/sections
    • Blank line before and after bullet lists
  • NO blank lines between bullet points in the same list
  • NO multiple consecutive blank lines (max 1 blank line between elements)
  • The output should be scannable and readable, even if the input wasn't

OUTPUT QUALITY:

  • Clean, scannable markdown that matches the original document's flow
  • Professional formatting that readers can quickly parse
  • All content intact - nothing added, nothing removed

TITLE FORMATTING RULES (CRITICAL)

The "title" field MUST use the EXACT format: "[Job Title] - [Company Name]"

  • Always use a HYPHEN (" - "), NEVER "at"
  • Example: "Senior ML Engineer - OpenAI" (CORRECT)
  • Example: "Senior ML Engineer at OpenAI" (WRONG — never use "at" in the title field)
  • Clean up the raw title: fix capitalization, remove company name if already embedded, remove location info
  • Keep the job title portion concise but don't lose important qualifiers (e.g., "Senior", "Lead", "Staff")
  • The "metaTitle" field uses a DIFFERENT format ("[Job Title] at [Company]") — do NOT confuse the two

SALARY DISPLAY FORMAT (IMPORTANT)

Format salary display as compact "k" notation:

  • $187,000 - $220,000 → "$187k - $220k"
  • $100,000 - $120,000 USD → "$100k - $120k"
  • €80,000 - €100,000 → "€80k - €100k"
  • Always use "k" for thousands, remove ".0" decimals
  • Include currency symbol at the start, not the end

WORK MODE INFERENCE RULES

Determine workMode using these rules IN ORDER:

  1. If explicitly stated ("remote", "hybrid", "on-site", "in-office") in title/description → use that
  2. If location contains "Remote" → REMOTE
  3. If multiple countries/continents listed → REMOTE (high confidence)
  4. If location is specific city only (e.g., "London", "New York", "Tokyo") with no remote mention → ONSITE
  5. If "flexible" or "work from home options" mentioned → HYBRID
  6. If unclear after all checks → null

EXPERIENCE LEVEL EXTRACTION RULES

Extract experienceLevel using these rules:

  1. From title keywords (highest priority):
    • "Intern/Internship" → ENTRY
    • "Junior/Jr/Associate" → JUNIOR
    • "Mid/Mid-level" → MID
    • "Senior/Sr/Staff/Principal" → SENIOR
    • "Lead/Tech Lead/Team Lead" → LEAD
    • "Director" → DIRECTOR
    • "Head of/Head" → HEAD
    • "VP/Vice President/Chief/C-level/CTO/CEO/CFO" → C_LEVEL
  2. From years requirement in description:
    • 0-1 years → ENTRY
    • 1-2 years → JUNIOR
    • 3-4 years → MID
    • 5+ years → SENIOR (unless title indicates higher level)
  3. If no indicators → null

LOCATION RULES (CRITICAL - keep compact)

Goal: Locations must be SHORT and COMPACT. Max 5 comma-separated values total.

  1. REMOTE JOBS:

    • If open to all/most regions → just "Remote"
    • If remote but limited to specific regions → use the CANONICAL REMOTE TOKENS below in parentheses
    • If remote limited to 2 regions → "Remote (US, Europe)" (still 1 badge)
    • NEVER list every individual country/city for remote jobs
  2. CANONICAL REMOTE TOKENS (use these EXACT values inside parentheses):

    • United States only → "Remote (US)"
    • US + Canada / North America → "Remote (NA)"
    • Europe / EU / EMEA / DACH / Nordics → "Remote (Europe)"
    • Asia / Asia Pacific / APAC / SEA / ANZ → "Remote (APAC)"
    • Latin America / South America / LATAM → "Remote (LATAM)"
    • Middle East / MENA / GCC → "Remote (MENA)"
    • Africa / Sub-Saharan Africa → "Remote (Africa)"
    • UK only → "Remote (UK)"
    • Specific country → "Remote (Singapore)", "Remote (Germany)", etc.
  3. ONSITE/HYBRID with specific cities:

    • For each city, ALSO include the country as a separate entry for discoverability
    • Example: "Dubai" → "Dubai, United Arab Emirates"
    • Example: "London, Berlin" → "London, United Kingdom, Berlin, Germany"
    • If 4+ cities in same country → use country name: "United States" and at most 2 key cities
    • NEVER include both city AND state AND country for the same place (e.g., "New York" not "New York, NY, United States")
    • But DO include city AND country as separate comma entries: "New York, United States"
  4. US-SPECIFIC:

    • NEVER list states separately (no "CA, NY, TX, FL...")
    • Many US cities → "United States" or top 2 cities + "United States"
    • Remote in US → "Remote (US)"
  5. MAX 5 comma-separated values in the locations field

TAG RULES (CRITICAL - content-derived skills only)

Goal: Tags must be SPECIFIC, ACTIONABLE skills/technologies/tools that appear IN THE JOB DESCRIPTION. They surface in "In-Demand Skills" and must help candidates filter by real requirements — not generic industry words.

FORBIDDEN TAGS (NEVER use these — they are useless on an AI job board):

  • "ai", "artificial intelligence", "machine learning", "tech", "product management", "business development", "marketing", "engineering", "design", "operations", "sales", "partnerships", "product marketing"
  • Any tag that merely restates the job category or industry (we already know it's AI/ML)
  • Any tag not explicitly mentioned or clearly implied by the job description text

REQUIRED: Tags must be EVIDENCE-BASED. Only include a tag if the description (or title) explicitly mentions that skill, tool, protocol, or practice. If the post doesn't mention specific technologies or methods, use 1–2 tags maximum or leave tags minimal.

GOOD TAGS (prefer these when present in the content):

  • AI/ML domains: "nlp", "computer vision", "reinforcement learning", "generative ai", "llm", "robotics", "autonomous systems"
  • Languages & runtimes: "python", "rust", "typescript", "go", "c++", "java", "cuda"
  • Frameworks & tools: "pytorch", "tensorflow", "jax", "hugging face", "langchain", "ray", "mlflow", "wandb"
  • Tools & practices: "figma", "wireframes", "product specs", "user research", "a/b testing", "sql", "react", "node"
  • Disciplines (only when clearly a focus): "ui/ux", "mlops", "data engineering", "compliance", "devrel", "deep learning"
  • Concrete deliverables/activities: "prds", "roadmaps", "model evaluation", "dashboards", "fine-tuning"

RULES:

  1. Do NOT repeat words already in the job title (e.g. "Product Manager" → no tag "product management").
  2. Prefer lowercase; well-known abbreviations are fine: "nlp", "ui/ux", "llm".
  3. Max 3 tags, prefer 2. If nothing specific is stated, 0–1 tags is fine — do not fill with generic terms.
  4. When in doubt, omit. Fewer accurate tags are better than generic ones.

RELEVANCY SCORING (for feed ranking)

Score how relevant this job is to an AI/ML job board audience (0-100).
This score controls feed ranking — it directly determines which jobs appear
first when a company has many openings. Be precise and consistent.

95-100: Core AI/ML research or engineering that BUILDS models
(ML Engineer, AI Researcher, NLP Scientist, Computer Vision, RL, LLM, MLOps)
80-94: Software infrastructure that directly SUPPORTS AI systems
(GPU/CUDA Engineer, ML Platform, Distributed Training Infra, AI DevOps)
65-79: Technical roles at AI companies (Software Engineer, Backend, Security, SRE)
50-64: Product, data, and design roles tied to AI products
(AI Product Manager, UX Researcher on AI tools, Data Analyst)
30-49: Business and go-to-market roles (Sales, Marketing, Finance, Legal, Recruiting)
10-29: Administrative and support roles (Office Manager, Coordinator, Compliance)
0-9: Non-technical facility/manual roles (Electrician, Custodian, HVAC, Security Guard)

Key heuristic: Would someone browsing an AI job board be excited to find this listing?
A "Senior ML Engineer" at Anthropic = 98. A "Facilities Coordinator" at Anthropic = 5.

AI RELEVANCE CLASSIFICATION (CRITICAL — controls whether job is listed)

Classify whether this job involves AI/ML/LLM/agentic technologies.
This is a strict binary gate for companies that aren't pure AI companies.

Set "isAIRelevant": true if ANY of these apply:

  • The role is fundamentally about AI/ML (researcher, ML engineer, NLP scientist, etc.)
  • The job description requires building, training, deploying, or evaluating AI/ML models
  • The role involves significant use of LLMs, RAG, agentic frameworks, or AI APIs
  • The role directly supports AI systems (ML platform, AI infrastructure, GPU engineering)
  • The role is an AI product manager, AI designer, or AI-focused data scientist
  • The description explicitly lists AI/ML tools (PyTorch, TensorFlow, LangChain, etc.)

Set "isAIRelevant": false if:

  • The role is generic software engineering with no AI/ML mention in the description
  • The role is business/operations/admin with no AI/ML component
  • The role is at a company that does AI but THIS specific position doesn't involve AI
  • Example: "Recruiter" at Stripe — no AI involvement even though Stripe uses AI
  • Example: "Backend Engineer" at Datadog — builds dashboards, no ML mentioned
  • Example: "Office Manager" at any company — never AI-relevant

Be strict: a job at an AI-adjacent company is NOT automatically AI-relevant.
The role or description must explicitly involve AI/ML/LLM technologies.

EXTRACTION RULES ===

  • Extract data ONLY if clearly stated
  • If not found, return null (not empty string, not "Competitive")
  • Use the structured data from job board when available (more reliable than parsing description)
  • For categoryId: ALWAYS select best matching category based on title and description

AVAILABLE CATEGORIES (id:name):
1:Business Development, 2:Data, 3:Design, 4:Engineering, 5:Finance, 6:Legal, 7:Marketing, 8:Operations, 9:People Ops, 10:Product Management, 11:Research, 12:Sales, 14:Support

EXPERIENCE LEVELS: ENTRY, JUNIOR, MID, SENIOR, LEAD, DIRECTOR, HEAD, C_LEVEL

JOB LISTING TO ANALYZE ===

Title: Staff Software Engineer - ML Observability
Company: Datadog
Location: Boston, Massachusetts, USA; New York, New York, USA
Description:
The ML Observability team builds cutting-edge tools to monitor, explain, and improve AI systems in production, particularly those leveraging Large Language Models (LLMs) and generative AI. We provide robust, scalable observability for AI workloads, including drift detection and model evaluation, and behavior tracing, enabling customers to ship AI with confidence.

As a Staff Engineer, you’ll lead the development of new features and foundational capabilities within Datadog’s LLM Observability product. You will shape product direction, drive experimentation, and apply your deep understanding of both AI systems and software engineering to solve open-ended problems in the fast-moving AI landscape. Your work will directly impact how our customers monitor, troubleshoot, and optimize LLM-based applications in production.

Join us in building the foundational tools that make AI systems observable, understandable, and reliable in the real world.

At Datadog, we place value in our office culture - the relationships and collaboration it builds and the creativity it brings to the table. We operate as a hybrid workplace to ensure our Datadogs can create a work-life harmony that best fits them.

What You’ll Do:

  • Drive design and implementation of LLM observability features.
  • Ideate, prototype, and scale new product features to provide insights and drive improvements for generative AI systems
  • Work cross-functionally with other eng teams, product, UX, and applied science to iterate fast and find product-market fit
  • Develop and extend tools for tracing, evaluating, and debugging LLMs
  • Influence architecture decisions and mentor engineers to build resilient, high-performance systems
  • Stay close to customer pain points and use those insights to guide product and engineering priorities
  • Stay current with industry trends and advancements in machine learning and observability, driving innovation within the team

Who You Are:

  • You have a BS/MS/PhD in a Computer Science, Engineering or related scientific field or equivalent experience
  • Deep understanding of distributed systems and scalable backend architectures
  • Hands-on experience building and shipping LLM-powered or GenAI applications.
  • Understanding of model internals, inference pipelines, evaluation techniques, and prompt engineering
  • Ability to thrive in ambiguous, fast-changing spaces and have a product-oriented mindset
  • You’re excited to shape the next generation of AI observability tools from the ground up
  • Communicate clearly, think rigorously, and take pride in clean, maintainable code
  • Experience with observability tools/platforms

Datadog values people from all walks of life. We understand not everyone will meet all the above qualifications on day one. That's okay. If you’re passionate about technology and want to grow your skills, we encourage you to apply.

Benefits and Growth:

  • Get to build tools for software engineers, just like yourself. And use the tools we build to accelerate our development.
  • Have a lot of influence on product direction and impact on the business .
  • Work with skilled, knowledgeable, and kind teammates who are happy to teach and learn
  • Competitive global benefits
  • Continuous professional development

Benefits and Growth listed above may vary based on the country of your employment and the nature of your employment with Datadog.

#LI-Hybrid

Datadog offers a competitive salary and equity package, and may include variable compensation. Actual compensation is based on factors such as the candidate's skills, qualifications, and experience. In addition, Datadog offers a wide range of best in class, comprehensive and inclusive employee benefits for this role including healthcare, dental, parental planning, and mental health benefits, a 401(k) plan and match, paid time off, fitness reimbursements, and a discounted employee stock purchase plan.

The reasonably estimated yearly salary for this role at Datadog is:

$234,000—$300,000 USD


About Datadog:

Datadog (NASDAQ: DDOG) is a global SaaS business, delivering a rare combination of growth and profitability. We are on a mission to break down silos and solve complexity in the cloud age by enabling digital transformation, cloud migration, and infrastructure monitoring of our customers’ entire technology stacks. Built by engineers, for engineers, Datadog is used by organizations of all sizes across a wide range of industries. Together, we champion professional development, diversity of thought, innovation, and work excellence to empower continuous growth. Join the pack and become part of a collaborative, pragmatic, and thoughtful people-first community where we solve tough problems, take smart risks, and celebrate one another. Learn more about #DatadogLife on Instagram, LinkedIn, and Datadog Learning Center.


Equal Opportunity at Datadog:

Datadog is proud to offer equal employment opportunity to everyone regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity, veteran status, and other characteristics protected by law. We also consider qualified applicants regardless of criminal histories, consistent with legal requirements. Here are our Candidate Legal Notices for your reference.

Datadog endeavors to make our Careers Page accessible to all users. If you would like to contact us regarding the accessibility of our website or need assistance completing the application process, please complete this form. This form is for accommodation requests only and cannot be used to inquire about the status of applications.

Privacy and AI Guidelines:

Any information you submit to Datadog as part of your application will be processed in accordance with Datadog’s Applicant and Candidate Privacy Notice. For information on our AI policy, please visit Interviewing at Datadog AI Guidelines.

Share job

Want to learn more about how the process works?

Learn more about Underprompt and how the application process works.

About Underprompt
Apply at Datadog
Visit job page →

Explore more jobs

More from Datadog

Similar roles you may like