This is a pretty interesting trend – as lots of processing becomes compute bottlenecked by shitty queries, an on-demand service like Dataproc really appealing. The trends as outlined in the post involve:
- reducing complexity
- resource isolation (so one query can’t kill everyone)
- better auditing & monitoring (so you know who to yell at)
- and more flexibility (so a select few can play around with few consequences)
TLDR: Lets trade off some performance across the board to better handle “lots of people writing lots of shitty queries”.
Also Ryan Noon sighting. Always asking the tough questions.