Finally !! You can query ANY dataset on 🤗Hugging Face with DuckDB 💛💛💛

It's now EASY to access datasets in SQL, e.g.

```
SELECT * FROM 'hf://datasets/<repo-id>/<file>'
```

Here I queried the dataset used for Cohere For AI Aya 101 or the new ✨Aya 23 ✨ multilingual LLMs:

blog.getwren.ai/how-to-load-hu…

Yesterday, @DuckDB and Hugging Face announced support for the hf:// path to 150,000+ datasets on Hugging Face. Today, we also added support for the latest DuckDB version on WrenAI, allowing you to load datasets directly from Hugging Face and use natural

🦆 ✖ 🤗
DuckDB 0.10.3 natively supports Hugging Face datasets!

Why does it matter?

Because it unlocks new use cases, possibly the one you need!

⬇️

DuckDB is a very interesting project. Its C++11 implementation could use some modernizing and a bit less allocations, but the idea very interesting.
shell.duckdb.org

Spark still makes sense on the cloud but DuckDB is king locally.

docs.coiled.io/blog/tpch.html

Also DuckDB has built in cosine similarity and other functions which is pretty neat

'With DuckDB as a browser for the data cloud, relational datasets are always just a hyperlink away' nikolasgoebel.com/2024/05/28/duc…

Great shout out to Ibis Project at the DuckDB talk at #PyConUS2024 PyCon US by Alex Monahan

#BetterTogether

My defacto tool for all small scale data analysis. The only missing feature is a good coupled plotting tool. DuckDB

duckdb as a 'browser for the data cloud' is a concept I can get behind nikolasgoebel.com/2024/05/28/duc…

New blog post: Access 150k+ Datasets from Hugging Face with DuckDB

This blog post, co-authored by the Hugging Face and DuckDB teams, describes how you can use the hf:// prefix in DuckDB to access datasets in Hugging Face repositories.

Read more at duckdb.org/2024/05/29/acc…

Pake multi-shot prompting untuk generate SQL.

Supaya hasil generasi gak keluar dari pakem SQL di database (DuckDB) maka cara paling aman adalah dengan memberikan contoh perintah SQL (contoh: INSERT)

New blog post on monitoring LLM apps with structlog + DuckDB dropping soon 👀

TidierDB.jl ❤️ DuckDB

TidierDB lets you run queries lazily in DuckDB without instantiating the results in memory (like dbplyr + duckdb). You can even reuse and build on top of the queries by using the `from_query()` function.

Traditional database systems require a predefined table schema and a subsequent data import process when working with CSV data.

In contrast, #DuckDB allows for direct reading of CSV files from disk, eliminating the need for explicit table creation and data loading.

Wrote up a one-pager on using DuckDB (or really any OLAP, local or not) to query traces for statistical inference. E.g. measuring the mean difference between two random variables. I'll try to show a multi-level example at some point.