Using the frontier models of the major AI labs can make projects prohibitively expensive, Fortunately many projects do not in fact require their capabilities and we can replace them with much smaller open source models trained on affordable local hardware or cloud compute. I can help if you’re looking to find ways to make projects which incorporate AI models more sustainable by using open source models.
Hoagy Davis-Digges
FacultyI teach open web data at Newspeak House, looking at tools to find and extract data from the internet.
AI tooling has made it possible to construct a much wider range of tools built on top of unstructured dataset, part of my current work is focused on using these tools to build products based on continuously updating data across large sets of different website.
I have...
- Worked in parliament as a researcher
- Worked as a developer for 38 Degrees
- Worked as data manager for the English National Opera
- Cofounded a legal NLP startup
- Built and currently maintain workincharities.co.uk
I’m at Ration Club fairly regularly, and will often arrange to discuss something informally there. I plan to run one session per term. If you wish to cover something in further depth then please contact me and we’ll arrange a time to meet 1-1.
Topics
Projects using AI as part of a process are often broken down into stages in order to analyse the behaviour of complex and non-deterministic elements of a program, as well as to allow it to be distributed across different hardware. I have experience working with building these pipelines and can advise on development.
While in theory we can gather the internet we need from the open web, in most cases we are constrained by the compute resources we have access to. One way round this is looking at some of the aggregated resources available and looking at how we can use these and apply them to our problem.
I’ll go through some of the large publicly available resources that it is worth being aware of, looking at how we access them, and some worked examples of using them in real projects, in particular how we can combine different datasets to unlock answers to a problem we’re trying to solve.
Getting useful data out of the internet often requires processing raw text and turning it into something usable. I can help explain how modern AI tooling makes this possible, how to use a standardised interface to extract structured data using either larger model providers or open source alternatives.