The end result is around 5 thousands `zstd` compressed Parquet files, taking less than 500MB in disk. 💫 The original CSVs take around 200GB and took some tinkering time to be able to download them all efficiently in only 3 to 4 hours.
Worked on a small side-project during the weekend to export ALL tables from The Spanish National Statistics Institute. - Code: github.com/davidgasquez...huggingface.co/datasets/dav...
I remember watching this talk and being blown away. Probably, the one with the biggest impact on my career, alongside the DDIA book!
The current public Jetstream instance has been running with `zstd` compression support for ~14 hours now, checking in on performance. The instance supports ~12 consumers right now with varying filters, but that fluctuates throughout the day. All consumers are currently caught up with live.
Published the second issue of "The Data Collective" newsletter. thedatacollective.substack.com/p/tdc-issue-2 A place to share some news, articles and interesting resources around Open Data that I'm continuously curating.
Better LLM models, more interesting browser tools and some ideas about data markets and common digital infrastructure.
While tinkering with a side project I ended up putting Dagster on a @huggingface.bsky.socialdavidgasquez-datalia-etl.hf.space
Big news: I'm working with @chris.bluelearning.oreilly.com/library/view... (O'Reilly Learning subscribers only at this point) and we're hoping to finish it next year.
Data is at the center of many challenges in system design today. Difficult issues such as scalability, consistency, reliability, efficiency, and maintainability need to be resolved. In addition, there...
Thanks! Lots of this is inspired by all your awesome and public work on Provider Quest! 🙌