One of the things that I’ve discovered about parenting is that babies are tracked religously. How much are they sleeping? Are they eating enough? Did they pee/poop the correct number of times in the last 24 hours? This isn’t just frettful new parent stuff (although it totally is that too) but things that pediatricians want to know and could be the early signs of problems when there isn’t a good way to know if there is a problem.
Yesterday I went to a data engineering meetup hosted by Roblox. The talk in the meetup was by Yan Shen and William Ng on how they cut down on processing costs in their data processing pipelines by making use of datelist tables.
A datelist table acts as an intermediate incremental accumulating aggregate of a quantity from a fact table. Their key feature is that they have a column that contains an array or map of dates where the this quantity was observed.
Recently, one of my colleagues came to me with a mysterious problem they were having with one of our production tasks running on AWS Batch. This particular task would be launched based on user interaction in a internal web application and run a fairly hefty machine learning model in docker container. Recently, this job started failing, not all the time, but sometimes on larger input files the job would fail with an out of memory error.
I’ve been working on an expanded AWS SDK for Pharo. Currently, 234 AWS services are available. See the code on GitHub.
Amazon Web Services (AWS) is a huge set (approximately 300 at time of writing) of services for doing just about anything related to computing infrastructure and tools, often with multiple ways to achieve the same or similar thing. The awesome thing about AWS is that there is an extensive set of APIs for working with these services that make it a coders paradise since it opens up great avennues for automation.
I’m constantly learning new things about the Python language. I consider myself a pretty good python programmer but often you never need to use all of the language features when writing your own code. For example I’ve not used is the class factory pattern using the type built in function. I’ve been aware of class factories, and read a few blog posts but never grokked it, until now.
A class factory is a function or another class that can create classes at runtime rather than you writing out the class definition in code.
Filing taxes in America sucks. Your options are to do it by hand, pay someone like Intuit, or if you are below a certain income threshold get some tax software for free. The kicker is that free tax software is from Intuit who will try very hard to make sure that you either don’t find it in the first place, or try to get you to pay for it and upsell you on something that should be free.
Smalltalk syntax can be a little confusing coming from other languages. Here I’ll show some comparisions between Python string operations and Smalltalk.
Substrings / Slicing Python strings use the slice notation where you can place up to three colon-separated values for the start, stop, and step. Python strings are 0-indexed and the stop argument is one past the final element that you want.
s = 'abcdefg' s[1:] # bcdefg s[:2] # ab s[1:6] # bcdef s[1:-1] # bcdef The slice notation in Python is compact and versatile to be used when getting the beginning, middle, or end of a string.
Starting out with Smalltalk can be a little jarring as it doesn’t have the similar syntax as launguages that are more heavily inspired by C. Dictionaries are one kind of data structure where I noticed this the most so I put together my notes on using them in Pharo with some comparisons to Python. In many other languages there is a subscript operator that allows you to access a value in a dictionary (and also a position in an array).
One of the key differences between SQL databases and graph databases is the concept of joining information from different nodes. In a tinkerpop-enabled graph database nodes have labels that define their type and properties that are part of that type. It’s natural to draw the comparison to a label being an SQL table and the properties of the nodes being the columns of that table. But trying to extend that analogy to compare joining in SQL is a little murky.
The first time I heard about Smalltalk was reading through the wikipedia page for Ruby, which mentioned it as an influence. At the time I was just a few months into my transition from a wet-lab biologist into a bioinformatician and trying to decide between Perl, Python, and Ruby as a scripting language to learn. Python became my language of choice after a long battle with Perl (this was some years ago and Perl was much more relevant).