Columnar file formats like Parquet/ORC are ubiquitous. Our VLDB paper with Xinyu Zeng + Huanchen Zhang + Wes McKinney studies their internals.

TLDR: They're not optimized for modern hardware. Something new is needed.

Paper: vldb.org/pvldb/vol17/p1…
Code: github.com/XinyuZeng/Eval…

Somebody tipped me off that a 2022 paper out of Saudia Arabia blatantly stole our entire 2019 ICDE Bulletin survey paper on using ML automatically optimize databases.

+ 2022 Plagiarism: eajournals.org/ejcsit/vol10-i…
+ 2019 Original: db.cs.cmu.edu/papers/2019/pa…

My #1 PhD student Matt Butrovich successfully completed his PhD defense. Thanks to the committee (@pateljm billions of packets Sam Madden). Matt's thesis is on accelerating databases with eBPF (telemetry, proxies, OLTP stores).

You have 60 days to hire him. Expect fierce competition.

The SIGMOD Jim Gray Dissertation Award has started receiving nominations! Deadline: March 15th, 2024. For more information please visit sigmod.org/sigmod-awards/…

CfP: Data Management on New Hardware Workshop, co-located with SIGMOD/PODS 2024 in Santiago, Chile. Papers due: March 15, 2024. Carsten Binnig and I are looking forward to your submissions to this *** 20th special edition of DaMoN ***! #DAMON2024 #SIGMOD2024 damon-db.org

Go Badgers!

'I had this crazy idea that I’m going to build a database engine that does not have a server, that talks directly to disk, and ignores the data types, and if you asked any of the experts of the day, they would say, “That’s impossible. That will never work. That’s a stupid idea.”

I'm back again with my annual retrospective of the last year in the world of databases. Major highlights include vector databases, @MariaDB problems, SQL:2023, the FAA database crash, and the most expensive password change ever: ottertune.com/blog/2023-data…

Last month the 'DB Research Meeting' was held @mitcsail, hosted by Sam Madden and Natassa Ailamaki (🙏). An encounter of the who's who in data systems research.

I warned for the declining impact of DB research & pitched better incentives for system work: bit.ly/dbmeeting-boncz

Overpublishing puts enormous stress on students and PIs.

And brings tons of money to publishers in STEM.

A new study shows that the number of papers is increasing FASTER than the number of #PhD graduates.

It’s an amazing work with very useful statistics. Huge kudos to the

New blog post — Voltron Data Update: Transitions

wesmckinney.com/blog/voltron-d…

Important stuff here 👇 It's great to see this transformation of the data stack gaining steam (and raising more capital) but much work still remains

Some long form thoughts on composable data management systems (and ApacheArrow, pandas, and more) on the heels of VLDB 2023 (also migrated my blog to Quarto!):

wesmckinney.com/blog/looking-b…

'My Favorite Bits of OSDI/ATC'23', new blog post looking at some trends, some papers I enjoyed, and a few random themes from this week's conference: brooker.co.za/blog/2023/07/1…

Join us on 6/19 for the 19th International Workshop on Data Management on New Hardware (DaMoN) SIGMOD/PODS 2024 !

- Invited talks by Frank Hady (Intel) & Huanchen Zhang (Tsinghua Univ)

- 17 exciting papers

See you in Seattle! Norman May #DAMON2023 #SIGMOD2023

damon-db.org

Thanks for this hot take dude who doesn't know the 60 year history of databases. H/T Prem Viswanathan

Mike and I have a WIP paper that analyzes all the (failed) attempts to replace SQL + relational model. This tweet has motivated me to finish it and submit it.

arxiv.org/abs/2304.05028 An Empirical Evaluation of Columnar Storage Formats. (arXiv:2304.05028v1 [cs.DB]) #databases

Great read, this quote is particularly insightful: “To put it another way, new technologies that require throwing away old technologies are harder to scale than new technologies that somehow bring along old technologies.”

Why is PACMMOD using single-column ACM style, and requiring SIGMOD/PODS 2024 camera-ready papers to reformat last minute? This is wrong at so many levels.

On Jeopardy! yesterday, the students all failed to answer the most important question in the history of the show.
HT Matt Butrovich for video!