Technical Lead, Data Engineering
Engineering – Technical Lead, Data Engineering – Washington, DC Preferred – Full-Time
DNS and netflow telemetry lands in a data lake in our own AWS account, detection pipelines score it, and the findings are written into an intelligence graph. Security researchers design the detections and often build them. This team builds everything around them: the pipelines that reduce lake data to what a detector needs, the packaging that lets it run on our platform, the orchestration that runs it on schedule at full volume, and the path that writes what it found back to the lake as evidence.
Our stack is Amazon EKS, Spark, Apache Iceberg and Delta Lake, Argo Workflows, Argo CD, Crossplane, and Python and Scala. We do not expect you to have used all of it. We do expect that you have worked in each of these areas, in whatever tools you used at the time: continuous integration and delivery, infrastructure as code, distributed data processing, container orchestration, a data lake on an open table format, and Python.
This role owns the schemas, the graph data model, and the detection framework the rest of the platform is built on. You will also write detections yourself. This is not a pure architecture role.
Washington, DC Preferred
Full-Time
Remote
Responsibilities
- The schemas. Define the canonical table structures the whole platform reads and writes, starting with the evidence model every detection writes to and every consumer reads from. Getting it wrong is expensive in a specific way: a finding written without a record of who produced it, when, and what it was based on cannot be repaired afterwards. You also own the detection type catalog — what each detection asserts, how it is expressed, and how long it stays relevant.
- The graph data model. What becomes a node, what becomes an edge, what context lives on each, how a new detection type is added without breaking anything already written, and how evidence ages out over time. The design language is settled and documented; the open questions are real and you will be answering them.
- The detection framework. Every detection needs the same four things around it: data reduced to its input grain, a container contract it conforms to, a workflow that runs it, and a path that writes its output back as evidence. You build that once, as shared code, so detections written by several people over several years do not each solve those four problems differently. This is the difference between four pipelines and forty.
- The interface with security research. Define what a detector receives, what it must emit, and what it may assume about the data it is fed. Researchers build against that contract, engineers build the pipeline around it, and you are responsible for whether it holds at production volume.
- Detections. You will build on the framework you design, not only specify it.
Required Qualifications
- Production Spark against an open table format, including table versioning. We run both Apache Iceberg and Delta Lake.
- You have designed a schema that other teams consumed as a contract, and lived through evolving it afterward.
- You have modeled data for a query workload, not only for storage. What the access patterns are, and what the model costs you when they change.
- You have written a specification that someone outside your team implemented against, and been responsible for whether what came back was acceptable.
- You have built internal libraries or frameworks that other engineers built on top of.
- Working experience across continuous integration and delivery, infrastructure as code, distributed data processing, container orchestration, a data lake on an open table format, and Python.
- Must be a US citizen
Preferred Qualifications
- Our specific stack: Amazon EKS, Argo Workflows, Argo CD, Crossplane.
- Scala, alongside Python.
- Graph data modeling, including how entities and relationships age out of a store.
Why Join Us
- Fully remote within the United States, with flexible working hours
- Foundational design ownership over the models the whole platform depends on
- A modern stack and an engineering culture that values automation and best practices