• 2 Posts
  • 10 Comments
Joined 1 year ago
cake
Cake day: June 27th, 2023

help-circle




  • I haven’t used them in Spark directly but here’s how they are used for computing sparse joins in a similar data processing framework:

    Let’s say you want to join some data “tables” A and B. When B has many more unique keys than are present in A, computing “A inner join B” would require lots of shuffling if B, including those extra keys.

    Knowing this, you can add a step before the join to compute a bloom filter of the keys in A, then apply the filter to B. Now the join from A to B-filtered only considers relevant keys from B, hopefully now with much less total computation than the original join.








  • I agree with how you characterized it and the term “ai engineer” didn’t resonate with me as defined by the author. If such an engineer doesn’t need to know about the data involved (“nor do they know the difference between a Data Lake or Data Warehouse”) then I don’t think they will be able to ship an AI/ML product based on data.

    New titles can be helpful for sorting out different roles with some shared skillsets such as the distinction which emerged between Data Scientist and ML Engineer at some companies to focus the latter on shipping production software using ML.