Why static workload is insufficient and what I learned by comparing HNSWLIB and DiskANN using streaming workload Image by DALLE-3Vector databases are built for high-dimensional vector retrieval. Today, many vectors are embeddings generated by deep neural nets like GPTs and CLIP to represent data points such as pieces of text, images, or audio tracks. Embeddings…

How to use BigQuery GENERATE_TEXT remote function 10 min read · 13 hours ago “Everyone can code and do NLP analysis in BigQuery with SQL knowledge and a good prompt structure” [Photo by Adi Goldstein on Unsplash]Since I started working with the Google Platform, Google has not stopped surprising me with its…

Pet Project for Data/Analytics Engineers: Explore Modern Data Stack Tools — dbt Core, Snowflake, Fivetran, GitHub Actions. Photo by Gaining Visuals on UnsplashHere is a simple and fast pet project for Data/Analytics Engineers, who want to kick the tires on Modern Data Stack tools including dbt Core, Snowflake, Fivetran, and GitHub Actions. This hands-on experience…

When LLMs are used to evaluate qualities like the correctness, accuracy, or relevance of a piece of text, consistency is paramount. If an LLM exhibits inconsistent judgements, then its evaluations become unreliable and untrustworthy. If an LLM evaluates the reasoning quality of arguments, but contradicts itself by rating an invalid argument as more logically sound…

Transforming static plots into captivating narratives Photo by Teemu Paananen on UnsplashPlotly supports an excellent foundation for animated plots. I highly recommend their basic tutorial here. However, plotly animations are primarily set up to add another dimension to the visualization — usually time. This is fantastic for adding more meaning to a plot. Animation, however,…

How to avoid overfitting whilst training your neural network https://www.flaticon.com/free-icons/neural-network. title=”neural network icons.” Neural network icons created by Vectors Tank — Flaticon.Background What is Overfitting? Lasso (L1) and Ridge (L2) Regularisation Early Stopping Dropout Other Methods Summary So far in this neural networks 101 series we have discussed two ways to improve the performance of…

How Game Theory can help you with every day’s decisions Image made by the author using MidjourneySomewhere in an uncharted galaxy, you and your friend are being held prisoners by an enigmatic group of extraterrestrial beings. They promise to let you go if you beat them in one of their games. You sit at a…

How GPT models’ view on occupations evolved over time Word cloud showing the top occupations generated by GPT-4 when prompted with “The woman/man works as a …”. Image created by the author.Back in December of 2020, I began writing a paper investigating biases in generative language models with a group at the University of Oxford.…

How pattern detection and pattern exploitation might elevate each other to a new level Instead of benchmark optimization- and machine learning algorithms against each other, we should consider how they can strengthen each other [Photo by Wedding Dreamz on Unsplash]Although most of us don’t see it, optimization algorithms (OAs) are at work everywhere. They plan…