Book a strategy call
Back to case studies
Research Services, AI

Enginuity: Scaling Data Pipelines for an IP Intelligence Layer

How a market demand platform for intellectual property engaged a Data Engineer.

Challenge

Enginuity tells innovators where demand for their technology actually exists: which public funders, which investors, which parts of the landscape are already asking for what they have. That is a valuable claim to make, and it sets a correspondingly high bar for the data underneath it. A matching engine of this kind has to hold its quality as it grows: more funders, more investors, more technologies, every new dataset re-processed and re-scored without someone running it by hand, and every result explainable rather than merely produced. Enginuity had the model working and chose to build the engineering around it (automation, scale and verification) before the dataset grew large enough to make that work harder. That was the request.

So the request was for a single data engineer, and a specific kind. Ten years and more of architecting and operating large-scale data systems, with migrations of hundreds of millions of records completed without downtime and pipelines handling a billion events a day: Python and PySpark, Kafka, Airflow, BigQuery, REST interfaces, and real depth in data modelling and dimensional design. Enginuity has a CEO and an internal engineer, so this was never a gap in direction. It was a gap in one particular kind of experience that the product’s next stage depended on.

One thing mattered above everything else in screening: work done on live systems with large sets of data. Scaling a pipeline that is already serving a product is a different discipline from building one in a clean environment, because every change has to land without interrupting what is already running and without disturbing results customers rely on. Shortlisting went to engineers who had done exactly that with big data and could describe how. The engagement was confirmed 40 hours after the request opened.

1
Talent Secured
3+
Months
40h
Time-to-hire

Outcome

One data engineer, and a narrow request. Enginuity did not need a team, it needed a decade of experience with data at a large scale, applied to a pipeline it already had - automating what was manual, making the processing faster and cheaper as the dataset grew, and putting verification underneath results its customers make decisions on.

  • One request, matched in 40 hours.
  • Senior data engineering applied to the pipeline the product’s core claim depends on.
  • Specialist data capacity alongside a CEO and one internal engineer, rather than a permanent hire.
  • Three months of an engagement: automating the data cycle, scaling the processing behind it, and validating what it produces.

A Strategic, Multi-Phase Hiring Approach

Phase: Request

One request for a senior data engineer on Python, PySpark, Airflow and Kafka, scoped to automation, scaling and validation of an existing production pipeline rather than a new build.

Phase: Matching

Weighting went to engineers who had scaled and migrated live systems without downtime and without disturbing results already in use, and who could describe how, with 2 of 3 shortlisted reaching interview.

Phase: Interviews

The CEO and the company’s internal engineer interviewed together, with the match confirmed 40 hours after the request opened.

Phase: Engagement

The engagement opened with part-time workload and hour cap, as well as daily hour tracker, and monthly reporting.

Phase: Support

Three months in, with changes for workloads and hour caps requested and approved as tasks moved between automation, performance and validation work.