Connor Skennerton

Can we imagine something better than embedded SQL strings

c.skennerton@gmail.com (Connor Skennerton) — Thu, 12 Dec 2024 00:00:00 +0000

SQL is lingua franca of the data world. Every tool supports it or has some version of it. I’ve been exposed to Postres, Trino, Spark-SQL, Flink-SQL, MySql, MS-SQL and they all work the same way for basic things. Of course the real challenge is when things get more complicated, certain functions have different names, or don’t exist, and some query patterns that work well in one engine don’t in another. (And I don’t just mean between OLTP and OLAP engines) While these are minor annoyances, it is pretty amazing to have a declarative language that is mostly familiar across tools and environments so that everyone from a product manager to a backend engineer can query terabytes of data in a few seconds.

Sadly what isn’t amazing is all the extra tooling that has been built on top of this in the data engineering world. Data Engineering can’t just rely on SQL, we need to orchestrate queries and parameterize them in various different ways. This is often done in Python (e.g. Airflow, Dagster, Prefect) using some degree of templating, whether that is using jinja2, f-strings, or .format(). Doing this has drastic negative effects on developer experience becuase we loose even basic things like syntax highlighting, or more advanded features of static analysis.

Some IDEs have extensions that detect embedded SQL and apply better linting rules. (Shout out to Pycharm which does this and lets you select a wide range of SQL dialects.) But if you don’t have that you’re stuck looking at a very long string and trying to understand why, when it’s all stitched together, where the syntax error is.

SQL itself is not a language that is easily modularized. It is difficult to create chunks or “functions” as part of the query. I’ve seen SQL embedding in python as f-strings where whole parts of the query are generated by calling functions in the string parameters (those functions themselves taking parameters from the main python code). It’s impossible to read but it’s done because the original author was struggling to make something that was composible.

There are other APIs, ORMs wrap up SQL in more native looking code and some tools like Spark have a data frame API which provides python native access. I’ve found getting out of SQL and using the data frame API quite a productivity boost but…

Because SQL is lingua franca it is difficult to on board users onto other ways of developing. Any open source Data or BI tool will accept SQL but ORM wrappers or data frame APIs aren’t supported. I try to work as much in Spark as possible and use the Dataframe API as much as I can due to it’s composibility, but it is hard to iterate quickly when Trino is just faster at some things and the only way to use it is via SQL through the company sanctioned web app BI tool.

Then again, maybe this doesn’t matter anymore: I can ask github copilot to take some SQL and convert it into pyspark Dataframe API, and it does a pretty good job of it

Helix Shortcuts

c.skennerton@gmail.com (Connor Skennerton) — Wed, 11 Dec 2024 00:00:00 +0000

Writing these down because I always forget…

Replace all occurances of something

create a selection. For example, using x for lines, v for a flexible selection, or % for the whole file
search in the selection with s followed by whatever you want to search for, then enter
(this will create a whole bunch of cursors) use c to change the text and enter insert-mode to replace

Résumé

c.skennerton@gmail.com (Connor Skennerton) — Sun, 10 Nov 2024 00:00:00 +0000

Experience

Senior Software Engineer

Roblox 2022 - Present

Part of the core Data Engineering team to build growth accounting and engagement datasets.
Developed tooling and frameworks to abstract boilerplate of building data pipelines.

Senior Staff Data Engineer

Pendulum Therapeutics 2019 — 2022

Worked cross functionally with relevant stakeholders at Pendulum to architect the full stack of software and data infrastructure.
Architected Python libraries to enable data scientists to easily interact with REST APIs and databases.
Managed junior software engineers, mentored their technical development with pair programming and code reviews while helping to balance their personal career objectives with company goals.
Led development of internal web applications. Creating the data model, API, and frontend layer.
Developed ETL and ELT pipelines for integrating Marketing, Manufacturing, and R&D data from SaaS applications and internal databases into a Snowflake data warehouse using Luigi, Prefect, and DBT.

Bioinformatics Cloud Engineer

Second Genome 2018 — 2019

Converted bioinformatics pipelines for processing genomic and metagenomic data into serverless, containerized applications using Docker, AWS Batch, AWS Lambda, AWS SQS.
Spearheaded python code consolidation within the company into a single unified library, enforcing test-driven development and software development lifecycle practices.
Developed ETL pipelines to automatically scrape NCBI and IMG databases and integrate them with private datasets into data warehouses

Postdoctoral Research Fellow

Caltech 2014―2018

My role was the head bioinformatician for a diverse group of microbiologists, chemists and data scientists at the California Institute of Technology (Caltech). My day to day tasks included the processing of genomic and proteomic data, and generating analysis summaries and plots using statistical software and programing tools such as R and python. As part of this role I was responsible for the production of automated pipelines for data analysis and have developed numerous tools and scripts for processing genomic data. I implemented SQL databases to store the extensive genomic and proteomic data that were generated and developed simple interfaces such that other members of the group could access this information without in-depth knowledge of SQL.

Graduate Student

University of Quensland 2010―2014

My PhD research focused on metagenomic analysis of microbial and viral communities from engineered ecosystems found in wastewater treatment plants. Throughout my PhD I was at the interface of microbial ecology, chemical engineering, and computational biology and have gained many skills ranging from microscopy, reactor operation and computer programming. I developed a number of bioinformatic pipelines and substantial computer software in perl, python and C++. My background in molecular biology allowed me to place the inferences of metagenomic data into a broader biological framework and work with other members of the lab that did not have computer skills.

Education

University of Queensland

2014, PhD in Microbial Ecology

2010, Bachelor of Biotechnology

Honors & Awards

2015, Dean’s award for outstanding PhD dissertation, University of Queensland

2010, Australian Research Council PhD Fellowship

Publications

McMurdie, P. J., Stoeva, M. K., Justice, N., Nemchek, M., Sieber, C. M. K., Tyagi, S., Gines J., Skennerton, C. T., Souza, M., Kolterman, O., Eid, J. (2022). “Increased circulating butyrate and ursodeoxycholate during probiotic intervention in humans with type 2 diabetes”. BMC Microbiol 22, 19 (2022).

Chadwick, G. L., Skennerton, C. T., Laso-Pérez, R., Leu, A. O., Speth, D. R., Yu, H., Morgan-Lang, C., Hatzenpichler, R., Goudeau, D., Malmstrom, R., Brazelton, W. J., Woyke, T., Hallam, S. J., Tyson, G. W., Wegener, G., Boetius, A., Orphan, V. J. (2022) “Comparative genomics reveals electron transfer and syntrophic mechanisms differentiating methanotrophic and methanogenic archaea”. PLoS Biol 20(1): e3001508.

Yu, H., Skennerton, C. T., Chadwick, G. L., Leu, A. O., Aoki, M., Tyson, G. W., Orphan, V. J. (2022) “Sulfate differentially stimulates but is not respired by diverse anaerobic methanotrophic archaea”. ISME J 16(1): 168-177.

Sim, M., Skennerton, C. T., Orphan, V. J. (2021) “Physiological, genomic, and sulfur isotopic characterization of methanol metabolism by Desulfovibrio carbinolicus”. PLoS ONE. 16(1): e0245069

Yu, H., Susanti, D., McGlynn, S. E., Skennerton, C. T., Chourey, K., Iyer, R., Scheller, S., Tavormina, P. L., Hettich, R. L., Mukhopadhyay, B., Orphan V. J. (2018) “Comparative Genomics and Proteomic Analysis of Assimilatory Sulfate Reduction Pathways in Anaerobic Methanotrophic Archaea”. Frontiers Microbiol. 9: 2917

Skennerton, C. T., Chourey, K., Iyer, R., Hettich, R. L., Tyson, G. W., Orphan, V. J. (2017) “Methane-Fueled Syntrophy through Extracellular Electron Transfer: Uncovering the Genomic Traits Conserved within Diverse Bacterial Partners of Anaerobic Methanotrophic Archaea”. mBio 8(4):e00530-17

Skennerton, C. T., Haroon, M. F., Briegel, A., Shi, J., Jensen, G. J., Tyson, G. W., Orphan, V. J. (2016) “Phylogenomic analysis of Candidatus ‘Izimaplasma’ species: free-living representatives from a Tenericutes clade found in methane seeps”. ISMEJ. 10: 2679–2692

Marlow, J. J., Skennerton, C. T., Li, Z., Chourey, K., Hettich, R. L., Pan, C., Orphan, V. J. (2016). “Proteomic Stable Isotope Probing Reveals Biosynthesis Dynamics of Slow Growing Methane Based Microbial Communities”. Frontiers Microbiol. 7: 563

Barr, J. J., Dutilh, B. E., Skennerton, C. T., Fukushima, T., Hastie, M. L., Gorman, J. J., Tyson, G. W., Bond, P. L. (2016). “Metagenomic and metaproteomic analyses of Accumulibacter phosphatis enriched floccular and granular biofilm”. Environ. Microbiol. 18(1): 273–287

Skennerton, C. T., Ward, L. M., Michel, A., Metcalfe, K., Valiente, C., Mullin, S., Chan, K. Y., Gradinaru, V., Orphan, V. J. (2015). “Genomic Reconstruction of an Uncultured Hydrothermal Vent Gammaproteobacterial Methanotroph (Family Methylothermaceae) Indicates Multiple Adaptations to Oxygen Limitation". Frontiers Microbiol. 6

Parks, D. H., Imelfort, M., Skennerton, C. T., Hugenholtz, P., Tyson, G. W. (2015). “CheckM: assessing the quality of microbial genomes recovered from isolates, single cells, and metagenomes”. Genome Res. 25: 1043–1055.

Skennerton, C. T., Barr, J. J., Slater, F. R., Bond, P. L., Tyson, G. W. (2015). “Expanding our view of genomic diversity in Candidatus Accumulibacter clades”. Environ. Microbiol. 17 (5): 1574–1585.

Soo, R. M., Skennerton, C. T., Sekiguchi, Y., Imelfort, M., Paech, S. J., Dennis, P. G., Steen, J. A., Parks, D. H., Tyson, G. W., Hugenholtz, P. (2014). “An expanded genomic representation of the phylum cyanobacteria”. Genome Biol. Evol. 6 (5): 1031– 1045.

Haroon, M. F., Skennerton, C. T., Steen, J. A., Lachner, N., Hugenholtz, P., Tyson, G. W. (2013). “In-Solution Fluorescence In Situ Hybridization and Fluorescence-Activated Cell Sorting for Single Cell and Population Genome Recovery”. In: Microbial Metagenomics, Metatranscriptomics, and Metaproteomics. Ed. by Edward F. DeLong. Vol. 531. Academic Press, 3–19.

Skennerton, C. T., Imelfort, M., Tyson, G. W. (2013). “Crass: identification and reconstruction of CRISPR from unassembled metagenomic data”. Nucleic Acids Res. 41 (10): e105.

Skennerton, C. T., Angly, F. E., Breitbart, M., Bragg, L., He, S., McMahon, K. D., Hugenholtz, P., Tyson, G. W. (2011) “Phage encoded H-NS: a potential achilles heel in the bacterial defence system”. PLoS One, 6 (5): e20095.

Trying out Helix as my text editor

c.skennerton@gmail.com (Connor Skennerton) — Sun, 09 Jul 2023 15:00:00 -0700

Helix is a modal editor in the same vein as Vim which means that there are different “modes” where the same key strokes mean different things. The main benefit for a modal editor is that many complex actions can be handled with just the keyboard rather than using the mouse or menus. I’ve been a Vim user for over a decade so I was interested to see what a modal editor that doesn’t come from that lineage would look like. My motivation was the complexity of getting my Neovim config set up just right when working with language servers and tree-sitter. Helix had these features built in so I wanted to see if Helix could improve my productivity without having to mess around with a lot of plugins.

I installed helix using homebrew as recommended (brew install helix). Initally I was confused about the command I needed to run. Turns out that it isn’t Helix but hx. The documentation directed me to use the tutorial with hx --tutor and I’m glad I did as it presented the editing commands in a logical and easy way. I did want to skip ahead at some points as I was familiar with and there are many similarities but as you get further through the differences show up. You’ll want to keep a link to the keymap handy. The are basic similarities in the key bindings but in helix the grammar is often reversed. For example, instead of typing dw you would type wd. This small change is really interesting as you can see what you are selecting (and potentially modify it) before doing anything destructive. It’s sort of like having Vim visual mode always being on. Comands that normally move the cursor in vim will change the selection in helix. So, w won’t just move to the end of the word but instead select the whole word. For basic incantations like dw (in Vim) or wd (in Helix) this will have the same effect - the word will be deleted.

The muscle memory from Vim to Helix can be tough. In particular, I’m fond of typing x to delete a single character and gg (or G) to move to parts of the current file. Having these be different things in Helix took some getting used to. But not in a bad way. I enjoy Helix take on g command to let me move to all parts of the line or file. The mnemonic for g being “go” in Helix versus “global” in Vim seems more natural to me. I do miss marks though. A very common workflow for me was setting locations in Vim using ma or mb to jump around a buffer. In Helix, m is for entering “match mode” that lets you select different parts of the buffer. It’s very useful just hard to break habits. There currently isn’t an equivelent to vim’s marks in helix. Instead you can save locations in a jump list using Crtl + s and then returning to those locations using Space + j and choosing them from a pop up dialog. It’s certainly not as fast so I hope the helix developers consider having named marks in the future.

The other issue I ran into was the default python configuration requires pylsp which doesn’t work well with my setup that uses pyenv. Because I use per-project python environments but pyenv creates a shim in my path which Helix picks up and uses. The language server command needs to be in your path and I couldn’t figure out how to get it to work without installing pylsp into all of my environments. I ended up using pywright which I installed globally through npm. It’s not the cleanest solution but it works.

Installation

brew install helix npm
npm install --location=global pyright

Config

`~/.config/helix/config.toml`

theme = "noctis"

[editor]
line-number = "absolute"

[editor.cursor-shape]
insert = "bar"
normal = "block"
select = "underline"

[editor.file-picker]
hidden = false

`~/.config/helix/languages.toml`

[[language]]
name = "python"
scope = "source.python"
injection-regex = "python"
file-types = ["py","pyi","py3","pyw","ptl",".pythonstartup",".pythonrc","SConstruct"]
shebangs = ["python"]
roots = ["setup.py", "setup.cfg", "pyproject.toml"]
comment-token = "#"
language-server = { command = "pyright-langserver", args = ["--stdio"] }
indent = { tab-width = 4, unit = "    " }
# will get "Async jobs timed out" errors if this empty config is not added
config = {}

Building a personal baby tracker with Pharo and Seaside

c.skennerton@gmail.com (Connor Skennerton) — Mon, 29 May 2023 12:35:00 -0700

One of the things that I’ve discovered about parenting is that babies are tracked religously. How much are they sleeping? Are they eating enough? Did they pee/poop the correct number of times in the last 24 hours? This isn’t just frettful new parent stuff (although it totally is that too) but things that pediatricians want to know and could be the early signs of problems when there isn’t a good way to know if there is a problem. There are many apps that will help with this giving simple GUI to measure all sorts of things. Here is a website that compares some, and a quick search of the App Store will give many more. The problem I have with all of these solutions is that you are minutely tracking the first stages of your childs life and development and giving all that data to someone else. The people who make these apps are running a business and you’re the product. Now I don’t necessarily have an issue with that, I’m using multiple apps that work by targeting ads to me. I get it, but I didn’t want to feed the personal data mining machine. I didn’t want to read a privacy policy or even worry about it, I wanted something where there was no data to be given away and sold. So I wrote my own personal baby tracker.

The requirements were pretty simple:

Dead easy UI so even in a sleep deprived state we could log updates.
Available across multiple devices.
Only accessible to people caring for our newborn.
Easy development and deployment. This is a personal project so it doesn’t need to scale.
Have some fun.

I wrote an MVP that works for me using the Seaside framework with Pharo 10. This was my first time using Seaside and I was really impressed with how easy it was to generate pages. No need to learn a templating language like in Python or Ruby. Instead, in Seaside, you write the HTML structure using smalltalk code. There was a big learning curve for me because of some initial misconceptions about how Seaside works. It’s not just for creating web pages, but for creating web apps. In Django/Flask or other frameworks I’ve used the request contains all of the information and state isn’t saved. In Seaside, state is saved, I found that thinking about Seaside to how something like React or Vue are used; there is global state and this is saved for you between actions. The difference being that Seaside acts server-side.

Seaside doesn’t provide any persistence like Django or Ruby on Rails. In that regard it’s a bit more like Flask. Pharo has many libraries for persistence I chose ReStore as it looked easy to set up. And it was! ReStore has a simple API that mimics the collection API in Pharo but it also allowed for some interesting customizations. ReStore has a simple way to add translations for Smalltalk messages into SQL functions or expressions. I was able to add in sqlite3’s unixepoch function by using the asUnixTime message that is understood by the DateAndTime:

ReStore translateMessage: #asUnixTime toFunction: 'UNIXEPOCH(%1)' asSQLFunctionIntegerResult.
ReStore translateMessage: #asDate toFunction: ('date(%1)' asSQLFunctionWithResultClass: Date).

(BabyEvent storedInstances satisfying: [
    :each | each startTime > '2022-10-29' asDate ] )
    project: [
        :each |
        each type ||each startDate || each count || each durationInSeconds sum || each durationInSeconds average 
    ].

This kind of customization was increadably simple. I have no idea how I could have done this in something like Django (or whether such a thing is even possible).

One of the difficulties when using a more niche framework is the lack of documentation for all of the use cases. One top of that this was my first Seaside app, so trying to map my knowledge from Django took a little time. For example, I couldn’t figure out how to keep track of the database connection, or where to store such initialization information. After asking a few “stupid questions” I was pointed to using the WASession class to hold “global” information like the database connection. Here are a couple of links I was pointed to:

Though I found that the Reddit sample application for seaside is a great resource for looking at the session and the deployment.

For deployment I found this blog nice for not running nginx and I used Digital Ocean’s tool for creating the actual configuration (as well as their hosting to set up a small droplet)

About Me

c.skennerton@gmail.com (Connor Skennerton) — Tue, 15 Nov 2022 09:15:00 -0800

I am a Data Engineer currently working at Roblox. I received my PhD at UQ in the Australian Centre for Ecogenomics. Throughout my PhD I was at the interface of microbial ecology, chemical engineering, and computational biology and have gained many skills ranging from microscopy, reactor operation and computer programming.

During my PhD, I developed many data analysis skills, including statistical inference, programming, and developing novel bioinformatics algorithms. I further honed these skills as a postdoctoral researcher at Caltech developing a SQL database to integrate diverse data sources while mentoring PhD students in bioinformatics.

I transitioned to working in industry at Second Genome where I used my bioinformatics knowledge to implement scalable data analysis workflows in AWS. At Pendulum Therapeutics I was involved in all aspects of the software and data stack including full-stack development of internal web applications, ETL and ELT pipelines, data warehousing, third-party data integrations, deployment/devops on AWS, and reporting and analytics.

I’m fluent in Python, however I have written software in diverse languages including Javascript, Typescript, Perl, R, Smalltalk, Java, and C++. As part of my duties I regularly collaborate with members of the scientific team and with non-technical members of the company to understand user requirements and build solutions for laboratory and manufacturing teams. I am passionate about analysis automation, eﬀicient data processing, and believe strongly in version control, test-driven development, and literate programming techniques to ensure reproducibility.

Datelist tables at Roblox Data Engineering Meetup

c.skennerton@gmail.com (Connor Skennerton) — Thu, 29 Sep 2022 08:00:00 -0700

Yesterday I went to a data engineering meetup hosted by Roblox. The talk in the meetup was by Yan Shen and William Ng on how they cut down on processing costs in their data processing pipelines by making use of datelist tables.

A datelist table acts as an intermediate incremental accumulating aggregate of a quantity from a fact table. Their key feature is that they have a column that contains an array or map of dates where the this quantity was observed.

Consider a raw fact table like (but imagine that it’s huge and partitioned by day):

userid	date	quantity
1	2022-09-01	1
3	2022-09-01	5
1	2022-09-04	6
1	2022-09-05	5

A datelist table would convert this information into a summary like:

userid	fisrt_date	last_date	date_list	dt
1	2022-09-01	2022-09-05	{“2022-09-01”: 1, “2022-09-04”: 6, “2022-09-05”: 5}	2022-09-05
3	2022-09-01	2022-09-01	{“2022-09-01”: 5}	2022-09-05

In this case the table granularity is an individual user (one user per row) and contains a column recording all of the dates at which quantity has been recorded. This table also has a partitioning column dt that is used to record the date at which the rows were last updated. The table can also record other important attributes, in this case the first date and last date that the quantity was recorded.

The value in this structure comes from updating the information when new partitions are added to the original fact table. Let’s say that new data comes in that looks like the following:

userid	date	quantity
3	2022-09-06	9

We can then update the datelist table using only this new partition of the fact table:

userid	fisrt_date	last_date	date_list	dt
1	2022-09-01	2022-09-05	{“2022-09-01”: 1, “2022-09-04”: 6, “2022-09-05”: 5}	2022-09-06
3	2022-09-01	2022-09-06	{“2022-09-01”: 5, “2022-09-06”: 9}	2022-09-06

This can be achieved incrementally as the datelist table contains all of the history, so to generate the current datelist table we need only the previous days datelist table and the current partition of the fact table.

For small datasets, it’s possible to easily scan over the historical data, but as the dataset grows that becomes an unreasonable compute burden. In his talk, Yan and WIlliam gave the example of scanning over a raw fact table that had 10 TB of data generated per day. Scanning all the historical data every time a query was run required looking at petabytes of data. In comparison, using datelist table they only needed to scan through 10TB + ~0.5TB per day.

The datelist table can then act as an efficient intermediate table for calculating historical metrics like retention and totals of the quantity over the last 7 days, 28 days etc., from the table above it’s easy to aggregate the quantity over the previous 7 days:

userid	quantity_L7	dt
1	12	2022-09-06
3	14	2022-09-06

Tracking down strange out of memory errors in AWS Batch

c.skennerton@gmail.com (Connor Skennerton) — Sun, 28 Aug 2022 22:09:45 -0700

Recently, one of my colleagues came to me with a mysterious problem they were having with one of our production tasks running on AWS Batch. This particular task would be launched based on user interaction in a internal web application and run a fairly hefty machine learning model in docker container. Recently, this job started failing, not all the time, but sometimes on larger input files the job would fail with an out of memory error. My colleague came to me because they could run the same Docker container on their local machine without error so it seemed like an issue specific to the production environment.

To understand a little about AWS Batch: you create Docker containers and then specify a “job definition” that holds additional information about how to run that container, such as the amount of RAM, number of CPUs, environment variables, etc. This information can be overridden on a per job basis as well, which makes the system flexible in cases where there is a much large dataset to process. The jobs themselves are for batch processing, like taking input files and performing complex and timer consuming transformations on them. Jobs get submitted to a queue and the AWS Batch scheduler will attempt to pack as many of the containers onto one instance to minimize the amount of left over resources.

First up, I confirmed that the code did run without issue on an ec2 machine. And I got intrigued by the mystery because the ec2 machine definitely had less memory than the Batch job was set up to use. So why was the production job failing?

Okay, so how about just upping the memory in the Batch job? Can we just push this through to unblock our customers. By default, we run the production job with 64Gb of RAM, so I tried 128, 256, 512 Gb of memory… Nope, out of memory. Now there is no way that this job could be using that much RAM and no way that there is that much difference between running it on an ec2 and Batch (I mean, there should be none).

I looked at the ec2 dashboard and found that the instance trying to run the job was way smaller than expected, in fact it only had a 2 Gb of RAM. Wait, what. How come the instance that Batch launched was way smaller than the one that it would need to run the job. To double check I went back into the AWS Batch console, clicked on the individual job details and then strolled down to the container limit section. There I saw a helpful little exclamation mark saying that the way we passed in container overrides had been depreciated and that it was falling back to default values.

And so the mystery was revealed, the large RAM values weren’t ever being taken into account. It would have been nice if these jobs failed straight away instead of silently being accepted in a corrupted state.

Creating an AWS SDK for Pharo Smalltalk

c.skennerton@gmail.com (Connor Skennerton) — Sat, 02 Apr 2022 14:52:00 -0700

I’ve been working on an expanded AWS SDK for Pharo. Currently, 234 AWS services are available. See the code on GitHub.

Amazon Web Services (AWS) is a huge set (approximately 300 at time of writing) of services for doing just about anything related to computing infrastructure and tools, often with multiple ways to achieve the same or similar thing. The awesome thing about AWS is that there is an extensive set of APIs for working with these services that make it a coders paradise since it opens up great avennues for automation. The AWS APIs are implemented via HTTP which means that they can be accessed basically from any programming language that lets you send and receive web requests. However it’s much nicer to wrap up these API calls into software development kits (SDKs) that hide a lot of those details. AWS has a lot of officially supported languages and there are also some community efforts for other lanuages. Sadly, there isn’t a complete SDK for Smalltalk, although there is a incomplete version on Github that has partial support for some services. I wanted to expand on this but the task seems hurculean since there are thousands of APIs that need to be written.

How does AWS manage all its SDKs?

AWS is a big company but I still wanted to understand how they could release the same set of APIs in all these different programming languages and keep them all in sync. Thankfully, the SDKs are all open source so I looked through how the Python SDK, Boto3, is created. Boto3 actually relies on a separate package called botocore that handles all of the lower level HTTP API calls. If you’ve ever done boto3.client('batch') in Python, you’re actually using the code in botocore.

Botocore makes heavy use of code generation! The AWS APIs are defined in JSON formatted files and this data is read at runtime to dynamically create the correct classes. Looking at other languages, like Ruby or Go, there are the exact same files.

Aha! AWS doesn’t make these SDKs individually, they publish data files that enable code generation.

Understanding the data files

For each service there are multiple JSON files but the main one seems to be service-2.json. This file contains information about the “operations”, the API endpoints for the service, and the “shapes”, which describe the input and output data structures for the API requests. There is also a “metadata” section with lots of interesting things about the service itself.

{
  "version":"2.0",
  "metadata":{
    "apiVersion":"2020-08-01",
    "endpointPrefix":"aps",
    "jsonVersion":"1.1",
    "protocol":"rest-json",
    "serviceFullName":"Amazon Prometheus Service",
    "serviceId":"amp",
    "signatureVersion":"v4",
    "signingName":"aps",
    "uid":"amp-2020-08-01"
  },
  "operations":{},
  "shapes":{}
}

The metadata has a protocol key that broadly describes how to construct API requests and deserializing responses. There are five different protocols listed in the data files that AWS services use to construct queries: ‘json’, ‘ec2’, ‘rest-xml’, ‘rest-json’, ‘query’.

Protocol Name	Services Using Protocol
rest-json	120
json	114
query	19
rest-xml	4
ec2	1

Unsurprisingly, communicating with JSON is the most popular choice but it is split between json and rest-json. It would appear that the EC2 service is a one-of-a-kind and has it’s own dedicated protocol, which is probably due to it being one of the oldest AWS services.

JSON Protocol

This appears to be the simplest of the protocols. All of the requests are sent to the root path of the host and a header, x-amz-target, provides information about which operation to target. All of the required parameters are provided in the body of the request as JSON.

REST-JSON Protocol

Each operation has a different path on the host server. Input parameters can be placed in the URL path, query string, headers, or request body. The input shape is responsible for encoding which parameters go where and the request body is formatted with JSON.

Query Protocol

The input shape is encoded as x-www-form-urlencoded and added to the query string of the request. Nested information of the input shape, such as structures, maps, and lists are encoded are via an incrementally generated prefix so that the key in the query string could become something like Foo.bar.member.1=value for a shape that looks something like {"Foo": {"bar": ["value"]}} in JSON. Structures are created by prefix.member, lists are created by prefix.listName.1, and maps are created by prefix.mapName.key and prefix.mapName.value

REST-XML Protocol

The same as rest-json but the request and response bodies are formatted with XML rather than JSON.

EC2 Protocol

Very similar to the Query protocol; used only on the EC2 service.

Creating an AWS service code generator

The python an Ruby SDKs ship with the JSON files and read them every time there is a call to generate a new client. This is an interesting approach that makes use of metaprogramming in these languages. I chose instead to use code generation to build the services beforehand. Since Pharo doesn’t separate code and runtime once you create a class it’s created and can be accessed in the image so creating the classes at “runtime” doesn’t really mean the same thing as it does in those other languages. Furthermore, it’s better to generate the classes so they can be imported with metacello individually if needed. It’s very unlikely that you need all 300 odd AWS services in your image so there is no reason to get all of the data for them. Of course, you are free to use the code generation package yourself if you have the data files on hand and want to go that way.

Operations

I followed the general structure of other SDKs and created a single class for each service. That class has a number of messages, one for each of the operations. An operation is simply an API request.

Here is an example of how an operation is encoded in the JSON data files:

  "operations":{
    "CreateAlertManagerDefinition":{
      "name":"CreateAlertManagerDefinition",
      "http":{
        "method":"POST",
        "requestUri":"/workspaces/{workspaceId}/alertmanager/definition",
        "responseCode":202
      },
      "input":{"shape":"CreateAlertManagerDefinitionRequest"},
      "output":{"shape":"CreateAlertManagerDefinitionResponse"},
      "errors":[
        {"shape":"ThrottlingException"},
        {"shape":"ConflictException"},
        {"shape":"ValidationException"},
        {"shape":"ResourceNotFoundException"},
        {"shape":"AccessDeniedException"},
        {"shape":"InternalServerException"},
        {"shape":"ServiceQuotaExceededException"}
      ],
      "documentation":"<p>Create an alert manager definition.</p>",
      "idempotent":true
    },

The operation name, CreateAlertManagerDefinition, would get converted to the message AWSAmp>>createAlertManagerDefinition: aCreateAlertManagerDefinitionRequest.

Due to the way that messages work I chose to model the shapes as objects. In Python SDK, shapes are not turned into objects but instead the python function calls contain many keyword arguments. This works well for Python where keyword arguments can be given in any order to a function. In Pharo, arguments need to be given in order so it becomes quite cumbersome to put in 10 different arguments, most of which are optional.

Instead, the operations take a single argument, a request object, that can be serialized. This is the same approach that the Go SDK takes.

The operation above has a templated path on the server: the requestUri of the operation contains a parameter, workspaceId that must be obtained from the input. In this case the CreateAlertManagerDefinitionRequest is modeled as an object in Pharo that contains a workspaceId accessor.

The definition of the shapes is also in the JSON data definitions. Below is what the CreateAlertManagerDefinitionRequest looks like:

    "CreateAlertManagerDefinitionRequest":{
      "type":"structure",
      "required":[
        "data",
        "workspaceId"
      ],
      "members":{
        "clientToken":{
          "shape":"IdempotencyToken",
          "documentation":"<p>Optional, unique, case-sensitive, user-provided identifier to ensure the idempotency of the request.</p>",
          "idempotencyToken":true
        },
        "data":{
          "shape":"AlertManagerDefinitionData",
          "documentation":"<p>The alert manager definition data.</p>"
        },
        "workspaceId":{
          "shape":"WorkspaceId",
          "documentation":"<p>The ID of the workspace in which to create the alert manager definition.</p>",
          "location":"uri",
          "locationName":"workspaceId"
        }
      },
      "documentation":"<p>Represents the input of a CreateAlertManagerDefinition operation.</p>"
    },

Shapes also have types, in this case it is a structure but there are others like map, list, string, timestamp, etc. Structures have a members dictionary which for the smalltalk SDK get converted into the accessors of the object. You can also see that the members contain some metadata about where in the request they should be put. The workspaceId member has a location of the uri, whereas the other two members don’t have that information (based on the other SDKs this means to put them in the default location, meaning the body of the request). The members also define their own shapes creating a recursive descent till the shapes are basic types like string.

Using the code generator

"Load in the code generator group"
Metacello new
    baseline: 'AWS';
    repository: 'github://ctSkennerton/aws-sdk-smalltalk/pharo-repository';
    load: #('Client-Creator').

acc := AWSClientCreator new.

"point the code generator to the directory containing the AWS data files.
You will need to bootstrap them from another SDK like botocore.
"
serviceData := acc findJson: '/botocore/data' asFileReference.

"Load in the JSON definition for the Athena service and create the classes"
athenaDefinition := (NeoJSONReader on: (serviceData at: 'athena') readStream) next.
acc createFromJson: athenaDefinition.

Install and try it out

"Load only the Amp service from the example above.
Replace the parameter to load with the services you use"
Metacello new
    baseline: 'AWS';
    repository: 'github://ctSkennerton/aws-sdk-smalltalk/pharo-repository';
    load: #('Amp').


"default parameters to pass to the request. Change to your values"
workspace := 'CAJK12N'.
data := 'COMPLETE'.

"Create a new AMP service. Will use your credentials from ~/.aws"
service := AWSAmp new.

resp := service createAlertManagerDefinition:
    (AWSAmpCreateAlertManagerDefinitionRequest new workspaceId: workspace; data: data).

Project Status, Work Still to be Done

This is still very much a work in progress. So far the class generators cover the JSON and REST-JSON protocols, which is still 234 different services but lacks some important ones such as ec2, s3, and SNS. There also isn’t any nice parsing of responses from any of the services so the response objects are basically the raw JSON returned.

Contributions very welcome, make a pull request on GitHub.

Dynamic types in Python.

c.skennerton@gmail.com (Connor Skennerton) — Wed, 24 Nov 2021 01:45:00 -0800

I’m constantly learning new things about the Python language. I consider myself a pretty good python programmer but often you never need to use all of the language features when writing your own code. For example I’ve not used is the class factory pattern using the type built in function. I’ve been aware of class factories, and read a few blog posts but never grokked it, until now.

A class factory is a function or another class that can create classes at runtime rather than you writing out the class definition in code. In other words it’s a way to programatically create classes. Normally when you write python you create classes in your code like so:

class A:
    def __init__(self):
        self.x = 1
        self.y = 2


a = A()
print(a.x, a.y)

So above, we define the name of a class, A, and give it an initialization method. Then we create and instance of that class, a, and print its two instance variables. There is another way to create this class though using the type function, which lets you create types (classes). We could reconstruct the code above as follows:

def init_func(self):
    self.x = 1
    self.y = 2
    
A = type('A', (), {"__init__": init_func})

a = A()
print(a.x, a.y)

The fun bit is the third argument to type which defines a dictionary where the keys are the names of methods (and class variables) for the class. The values of the dictionary are either function names or values. Note that for init_func you don’t want to add the parentheses as this will call the function (and give an error). For me, this also helped illuminate why in python we have to pass self as the first argument to a method – because you can attach on any old function to a class. It also goes to show that self is just a convention. If you were so inclined you could make python look like another language that uses this and not self:

# Let's make python look a little more like C++
# which uses "this" instead of "self"
# Probably don't want to do this for real as it
# will really confuse people
def constructor(this):
    this.x = 1
    this.y = 2

B = type('Cpp', (), {'__init__': constructor})
b = B()
print(b.x, b.y)

So now lets use this knowledge to make our “factory”. Take most of the code above and wrap it in another function. This way you can pass configuration for your to-be-created class into the outer function. You can extend this as far as you want and don’t have to make it a function either, you could create a class whose job it is to create other classes – a class factory class!

def class_factory(name='A'):
    def init_func(self):
        self.x = 1
        self.y = 2
        
    return type(name, (), {"__init__": init_func})

A = class_factory()
a = A()
print(a.x, a.y)

This is all nice and abstract but how and why would you use this?

Lets have a look at the marshmallow library which performs object serialization. With marshmallow you can define incoming data, usually in the form of JSON and convert it to native python data types (and vice-versa) while validating the data conforms to a certain schema. Here is the example from their website:

from datetime import date
from pprint import pprint

from marshmallow import Schema, fields


class ArtistSchema(Schema):
    name = fields.Str()


class AlbumSchema(Schema):
    title = fields.Str()
    release_date = fields.Date()
    artist = fields.Nested(ArtistSchema())


bowie = dict(name="David Bowie")
album = dict(artist=bowie, title="Hunky Dory", release_date=date(1971, 12, 17))

schema = AlbumSchema()
result = schema.dump(album)
pprint(result, indent=2)
# { 'artist': {'name': 'David Bowie'},
#   'release_date': '1971-12-17',
#   'title': 'Hunky Dory'}

The key things to note are that you have to create schema classes that define fields that are of particular types. But what if you don’t know the types beforehand or the types changed outside of the control of your program? For example, you have a web service that accepts incoming data, but the input data is user defined (to a point) and the actual schema of the data are stored outside of your program. It’s not feasible for you to code in a new class in your web service every time the data changes but you also don’t want to accept any old user data. What you really need is to read in one of these external schemas to create a validator for the input data on the fly.

Class factories to the rescue! Lets see how we can build a marshmallow schema class from a simple definition stored in JSON as a dictionary of field names to types.

from typing import Dict
from marshmallow import Schema, fields

def schema_factory(schema_definition: Dict[str, str]):
    """
        The schema definition is a simple dictionary with the name
        of the field as the key and a string that defines its type
        as the value. For example:
        
        {
            'field1': 'str',
            'field2': 'date'
        }
    """
    fields = {}
    field_type_map = {'str': fields.Str,
        'date': fields.Date
        # add in other types as needed
    }
    for schema_field_name, schema_field_value in schema_definition.items():
        # We find the right marshmallow field and initialize it.
        # Fields then defines the class variables that you would 
        # manually do when defining a marshmallow Schema.
        fields[schema_field_name] = field_type_map[schema_field_value]()
        
    return type('CustomSchema', (Schema,), fields)

Open Source Tax Software

c.skennerton@gmail.com (Connor Skennerton) — Mon, 30 Aug 2021 18:45:28 -0700

Filing taxes in America sucks. Your options are to do it by hand, pay someone like Intuit, or if you are below a certain income threshold get some tax software for free. The kicker is that free tax software is from Intuit who will try very hard to make sure that you either don’t find it in the first place, or try to get you to pay for it and upsell you on something that should be free. Check out this excellent article from ProPublica that outlines Intuit’s tactics.

I’ve tried to do my taxes by hand and made mistakes that mean I’d have to amend my return. This tax year I was pleasantly surprised to find https://excel1040.com, which offers a fairly comprehensive list of federal tax forms. The beauty of excel1040 is that the forms themselves have been recreated in excel, one form per tab. It’s incredibly powerful to see the forms get filled out automatically as you put in input values from your W-2 or 1099 forms. And since it’s excel you can freely overwrite values if you think they are wrong or add in new logic if a form isn’t implemented yet. The author even says that you can print out the forms from excel and they’ll be accepted, though I didn’t try that and instead copied the values from the spreadsheet onto the pdf forms from the IRS.

Though I’m sure there are unsupported parts of the tax code in excel1040 it does have a wealth of different forms and the author appears to have been slowly evolving the complexity of the task over many years.

The only downside with excel1040 is that it’s tricky to understand how all of the formulas are implemented as there is some complex logic that has to be squeezed into a single line formula. Here are a couple of examples of what I mean:

=IF(P74<>"",P74,IF(NOT(F6251_PIII),"",IF(AND(SchDTW_Used,NOT(CGTW)),'Sch. D WS'!P87,IF(CGTW,'Line 16 (QDCG Tax)'!J34,0))))
=IF(AM44<>"",ROUND(AM44,0),IF(OR(AM25,AM29),"",IF(AL6<>"",IRA_CONV_YOU,IF(AL7<>"",IRA_CONV_SPOUSE))))

After I found excel1040, I also discovered https://ustaxes.org which is aiming to be a replacement for paid tax software. It’s a website and downloadable application that lets you add in your income and it does the math and fills in the IRS pdf forms automatically. It is written in typescript and felt more natural to me (as a programmer) to understand the logic. I’ve already contributed a couple of additions and plan to keep contributing. It doesn’t have as many of the forms implemented as excel1040 but I see it having at least one advantage: Because it is written in typescript as a custom application, we can take advantage of software tools like automatic testing, version control, and better structure to the formulas and logic.

I hope that efforts like excel1040 and ustaxes.org get more popular so that Americans have additional, free choices when filing their tax.

Using Strings in Pharo Smalltalk

c.skennerton@gmail.com (Connor Skennerton) — Mon, 19 Jul 2021 01:20:17 -0700

Smalltalk syntax can be a little confusing coming from other languages. Here I’ll show some comparisions between Python string operations and Smalltalk.

Substrings / Slicing

Python strings use the slice notation where you can place up to three colon-separated values for the start, stop, and step. Python strings are 0-indexed and the stop argument is one past the final element that you want.

s = 'abcdefg'
s[1:]  # bcdefg
s[:2]  # ab
s[1:6]  # bcdef
s[1:-1]  # bcdef

The slice notation in Python is compact and versatile to be used when getting the beginning, middle, or end of a string. Smalltalk achieves the same results using a number of different messages that can be a little confusing when you first get into it.

s := 'abcdefg'

"Leave off only the first character"
s allButFirst  "bcdefg"

"allButFirst with an argument to leave off that many characters"
s allButFirst: 2  "cdefg"

"There are also the first and last messages that produce the same result
as s[1:] or s[:5]. Remember though that Smalltalk's indexing is 1-based 
and inclusive"
s first: 2  "ab"
s last: 2  "fg"

"Get the middle of the string"
s copyFrom: 2 to: 6  "bcdef"

"Smalltalk indexing doesn't have the same trick with negative indexes,
instead you need to calculate the size"
s copyFrom: 2 to: s size - 1

Formatting

Python has extensive string formatting using either the .format() method of string or using f-strings. Pharo has a similar format: message that can be used for inpterpolating other objects in strings. Each of these objects must have a asString method for this to work.

"pass in a collection where values are indexed by number (like Array or OrderedCollection)"
'ab {1} ef {2}' format: {'cd'. 'gh'}.  "ab cd ef gh"

"pass in a collection where values are indexed by name"
'ab {one} ef {two}' format: 
    (Dictionary with: #one -> 'cd' with: #two -> 'gh').  "ab cd ef gh"

Unlike python there is no format mini-language. Which means that if you want to format things like numbers in a specific way you need to do that before passing the object to format:. Numbers have a basic message printString that will use the default settings. For very large or very small numbers it will print in scientific notation.

There isn’t a general number formatter like printf in C/C++, instead there are messages for each type of formatting you want on a number. printPaddedWith: aCharacter to: aNumber is used to perform left-side padding with a character of your choice. Normally you would choose zero to mimic the printf behaviour but you can choose any character you like.

123 printPaddedWith: $0 to: 8     "'00000123'"
123 printPaddedWith: $, to: 8     "',,,,,123'"
123.31 printPaddedWith: $0 to: 6  "'000123.31'"
123.31 printPaddedWith: $0 to: 1  "'123.31'"

printShowingDecimalPlaces: placesDesired is for controlling the number of decimal places.

123.31 printShowingDecimalPlaces: 12  "'123.310000000000'"

To combine these with the format: message you could do something like the following to make a dynamic array of the values you want:

'{1}' format: {  1.123 printShowingDecimalPlaces: 12. }

There isn’t a scientific formatter though, instead the default is to print a number in decimal format when it’s small, and switch to the scientific format as it gets bigger. I found this a little annoying as I wanted to print numbers in a consistent scientific format no matter their size.

There is a package called NumberPrinter that has more versatile printing options, including scientific printing. For example, the default settings with FloatPrinterScientificFormat would convert 123.123 to 1.23123e2. But you can change that with different messages to the number printer.

rws := ReadWriteStream with: String new.
fp := FloatPrinterScientificFormat new.
fp exponentDigitCount: 2.
fp exponentChar: $E.
fp print: 123.123 on: rws.
rws contents  "1.23123E02"

Sadly this is a bit more verbose than python: '{:.5E}'.format(0.09112346) = '9.11235E-02'

Using Dictionaries in Pharo Smalltalk

c.skennerton@gmail.com (Connor Skennerton) — Mon, 21 Dec 2020 13:54:32 -0800

Starting out with Smalltalk can be a little jarring as it doesn’t have the similar syntax as launguages that are more heavily inspired by C. Dictionaries are one kind of data structure where I noticed this the most so I put together my notes on using them in Pharo with some comparisons to Python. In many other languages there is a subscript operator that allows you to access a value in a dictionary (and also a position in an array). It’s often written dict[key] to access a value and dict[key]=val to set a value to an existing key or add in a new key. Smalltalk doesn’t contain a subscript operator and so interacting with dictionaries requires a slight change in the mental model to get things done.

Dictionaries in Pharo are composed of Associations. An Association is a class that holds a single key–value pair. You can create an Association by sending the -> binary message to an object (e.g. 1 -> 2). Associations can be used outside of dictionaries as well, for example the following code snippet creates an Array of Associations:

{ 1 -> 2. 3 -> 4. 5 -> 6 }

Initialization

You can create a Dictionary from an Array of Associations using the newFrom: message

Dictionary newFrom: { 1 -> 2. 3 -> 4. 5 -> 6 }.

or you can achieve the same thing with the at:put: message:

Dictionary new 
	at: 1 put: 2; 
	at: 3 put: 4; 
	at: 5 put: 6; 
	yourself.

While the newFrom: message seems intuative to me, the at:put: way of creating Dictionaries took me a while to get my head around. First, notice the semicolons between the successive calls to at:put:, this is for cascading messages that should all go to the same “receiver”, which in this case is the Dictionary object created by new. What happens if you don’t put in the semicolons? Pharo will get confused and think you are trying to send a single message called at:put:at:put:at:put: instead of three separate messages and you’ll get an error. Why is there no semicolon after new? So what’s happening is that the new message is creating an instance of the Dictionary class i.e. a Dictionary object which gets returned from the new message. The at:put: messages are then applied to the return value of new. If you put a semicolon after new then all of the subsequent messages will be sent to what new was sent to, rather than the result of new. In other words, at:put: will be sent to the Dictionary class and not a Dictionary object (which causes an error). The last bit of the statement, yourself, is needed to return the Dictionary object. Without yourself the return value of the final at:put: message is used, which is the value added to the dictionary. If the return value of at:put: was the dictionary object then the yourself message wouldn’t be needed at all.

Accessing elements

The basic method of getting values is to use the at: message:

d := Dictionary newFrom: { 1 -> 2. 3 -> 4. 5 -> 6 }.

d at: 3.  ">>> 4"

Just like in Python, if you try to access a key that doesn’t exist you’ll get an error

d at: 100.   "Error KeyNotFound"

but unlike Python there are multiple ways to avoid this error using variants of the at: message. The at:update:initial: message allows you either update or set a value in a dictionary:

d at: 1 update: [ :value | value + 1 ] initial: [ 1 ].

which is broadly equivelent to the following Python code

try:
    d[1] += 1
except KeyError:
    d[1] = 1

There are many other variants of at: that modify the behaviour depending on whether the key is present or not.

Enumerating

Enumerating in Pharo Smalltalk can also be achieved in multiple ways. The do: message is available in many classes for iterating, which for Dictionaries iterates through the values:

d do: [ :value | Transcript show: value; cr]

which is Python would be:

for value in d.values():
    print(value)

The do: message is an alias for valuesDo: and there is also a keysDo: for iterating through the keys and associationsDo: for iterating through the key–value pairs. Unlike Python which returns a tuple of the key and value, in Pharo an Association object is returned. This class responds to the key and value messages for accessing each.

d associationsDo: [ :pair | Transcript show: pair key; cr; show: pair value; cr]

and the Python equivelent:

for pair in d.items():
    print(pair[0])
    print(pair[1])

Despite the differences introduced through the syntax of each language, using dictionaries in Pharo Smalltalk and Python are basically the same.

Combining properites from different nodes in Gremlin

c.skennerton@gmail.com (Connor Skennerton) — Sun, 13 Dec 2020 15:14:10 -0800

One of the key differences between SQL databases and graph databases is the concept of joining information from different nodes. In a tinkerpop-enabled graph database nodes have labels that define their type and properties that are part of that type. It’s natural to draw the comparison to a label being an SQL table and the properties of the nodes being the columns of that table. But trying to extend that analogy to compare joining in SQL is a little murky. Insead of joining two tables together, it’s better to think about visiting different nodes and saving their information for later use. In gremlin you can achieve this in a few differnt ways but the most common is to use the as-step. The example below shows how to combine information using the “Modern” graph that is shipped with tinkerpop. Here there are two types of nodes, person and software; each have a “name” property. If you want a mapping of which people created which software you could do:

g.V().hasLabel('person').as('s').
    out('created').hasLabel('software').
    project('creator_name', 'software_name').
        by(select('s').values('name')).
        by('name')

In the example above the traversal starts with all of the people and saves the information in the variable, s, that’s what the as('s') is doing. It then travels to the software nodes, using out('created'). To get the information from both the software and the person I use the project-step, which creates a new structure. In this case it will be a mapping with creator_name and software_name as the keys. To get the creator name I use the select-step that recalls the information that was stored previously using the variable, s. The traversal above will produce a pair-wise mapping like so:

{creator_name=marko, software_name="lop"}
{creator_name=peter, software_name="lop"}
{creator_name=josh, software_name="lop"}

The other method of combining data is to use the group-step. In this case the returned structure will have a mapping based on how you group:

with the person as the key


g.V().hasLabel('person').group().
    by('name').
    by(out('created').hasLabel('software').values('name').fold())

or with the software name as the key

g.V().hasLabel('software').group().
    by('name').
    by(__.in('created').hasLabel('person').values('name').fold()).toList()

The first query gives you a dictionary where the persons name is the key and the values are a list of software names created by that person, like the following:

{
   marko: [lop],
   josh: [lop, ripple],
   peter: [lop]
}

The second version turns it around so that the software is the key and the creators are the list of values:

{
    lop: [marko, peter, josh],
    ripple: [josh]
}

Whether to use the query based on the as-step or the gorup-step is dependant on the structure that you want at the end and the performance. In my experience, grouping takes longer than using the as–project method and is more flexible too. You can save multiple steps in your traversal with different variable names and recall them at the appropriate time.

Exploratory data analysis with Pharo Smalltalk

c.skennerton@gmail.com (Connor Skennerton) — Thu, 10 Dec 2020 20:19:12 -0800

The first time I heard about Smalltalk was reading through the wikipedia page for Ruby, which mentioned it as an influence. At the time I was just a few months into my transition from a wet-lab biologist into a bioinformatician and trying to decide between Perl, Python, and Ruby as a scripting language to learn. Python became my language of choice after a long battle with Perl (this was some years ago and Perl was much more relevant). I remember being really intrigued at Smalltalk’s syntax and it’s pure object-oriented approach; I was just starting to userstand what these concepts meant but was already gravitating to OOP. This was a language that seemed very alien to what I was picking up in C-derived scripting languages. I learned that you made a file with your program in it and then passed that file as an argument to another program to interpret it — that’s just how programming worked.

In contrast, instead of code existing in files that are run by an interpreter or compiled into a binary program, Smalltalk has the concept of image-based persistence, which encapsulates the current program state and all of the code into one. Editing your code is equivelent to changing the program itself rather than a static copy that gets turned into your program. This is a pretty wild concept and makes programming much more interactive. An example of how this might streamline your workflow is in debugging. Often programmers will use a debugger to step through the running code line by line to see how variable change and find the place where are bug happens. When the bug is found you then have to go to the line in the source and make the required change (and either run the program again or re-compile). But with Smalltalk you’re running “live” code so modifying it during debugging fixes it once. While I can appreciate this now as a more accomplished programmer, as a newbie I was utterly confused by it. I never picked it up Smalltalk.

Recently I was back on wikipedia looking at some other programming language’s page and again there was that smalltalk influence. So I scratched the itch and finally downloaded Pharo — which seems to be one of the more popular implementations — and tried out some basic data visualization.

Pharo is an integrated environment, there isn’t really separate source code for writing programs; they get written one method at a time in the IDE. But there is a playground feature that allows you to write short scripts and generally play around. The code snippets I show below are all taken from the playground.

First we need to install some external packages. Unlike python or R there doesn’t seem to be a single source of packages, instead repositories can be downloaded from URLs:

Metacello new
    baseline: 'Roassal3Exporters';
    repository: 'github://ObjectProfile/Roassal3Exporters';
    load.

Metacello new
  baseline: 'DataFrame';
  repository: 'github://PolyMathOrg/DataFrame/src';
  load.

Now let’s look at some penguin data! I’m using the palmerpenduins dataset that was built with R in mind. The DataFrame package can’t load from URLs directly but Pharo provides some niffty built-in objects that allow us to convert the string URL into a ZnURL object, which we can retrieve the contents of (i.e. download).

"Convert the url string to a Url object and download the contents, passing that into the 
 data frame constructor.
"
df := DataFrame readFromCsv: 
    'https://raw.githubusercontent.com/allisonhorst/palmerpenguins/master/inst/extdata/penguins_raw.csv' asUrl retrieveContents .

After that we can then set up the plot using the various Roassal classes. In the example below I’m plotting the Culmen Length versus the Egg Date columns. It shows off a couple of tricks I had to learn: first, was how to plot a date on the axis; and second, was how to deal with NA values in the data frame.

chart := RSChart new.
lineaxes := RSScatterPlot new.

"Importing the CSV file doesn't recognize NA so convert them to NaN"
lineaxes x: ((df column: 'Date Egg') collect: 
            [ :d | d julianDayNumber ]) 
         y: ((df column: 'Culmen Length (mm)') collect: 
            [ :s | s = 'NA' ifTrue: ['NaN' asNumber] 
                            ifFalse: [s asNumber] ]).

chart addPlot: lineaxes.
vticks := RSVerticalTick new.
 
"Offset controls the horizontal position"
chart ylabel: 'Culmen Length (mm)' offset: -20.

chart addDecoration: vticks.
hticks := RSHorizontalTick new.
hticks numberOfTicks: 8.
"Convert the julian day numbers back to Date objects. The asString message will
be sent during rendering to make it look pretty"
hticks labelConversion: [ :x | Date julianDayNumber: x]; useDiagonalLabel .

chart addDecoration:  hticks.
chart openOnce.

Basic plot of Penguins with the date as the X-axis

Next I wanted to try and replicate a Facet Grid one of the best features of seaborn and ggplot2 for plotting small multiples. I couldn’t find anything built in that would be the equivelent but Roassal has all of the primatives to make this happen. My solution below is based on the examples in the Roassal package:

"Create a canvas to add our charts to"
c := RSCanvas new.
"First find the unique values in the species column and then loop through the values"
groups := ((df column: 'Species') uniqueValues) collect: [ :species | 

    "Subset the main dataset returning just the rows for the current species"
	spdf := df select: [:row | (row at: #Species) = species ].
    
    "Make a new group that will hold this chart"
	g := RSGroup new.
	chart := RSChart new.
	chart container: g.

    "Make a scatterplot"
    plot := RSScatterPlot new.
    
    "Get the x and y values the same as above"
	plot x: ((spdf column: 'Date Egg') asArray collect: 
            [ :d | d julianDayNumber ]) 
         y: ((spdf column: 'Culmen Length (mm)') asArray collect: 
            [ :s | s ifNil: ['NaN' asNumber] 
                     ifNotNil: [s asNumber] ]).
	chart addPlot: plot.
	chart title: species.
	chart build.
    "The final statement of a block is returned... so here
     we are converting the group object, which holds the chart
     into a shape, which is needed to add onto the canvas
    "
	g asShape .
	].

c addAll: groups.

"Layout the three groups horizontally on the canvas"
RSHorizontalLineLayout on: c shapes.
c @ RSCanvasController.
c openOnce.

Split out the into three charts, one for each species

As you can tell from the code samples above there is a lot of boilerplate to set this up. Roassal isn’t a ggplot clone, it’s much more low level like base plotting in R or matplotlib. It’s still impressive the variety of things that you can accomplish with it, 2D charts are just one aspect of Roassal, which is probably why there is a bit more boiler plate than you might want for just making a chart.

There is also a bit of roughness to the DataFrame package. The type detection isn’t as smart as in R so columns containing NA were being left as strings. I fixed this in my examples above wtih:

(df column: 'Culmen Length (mm)') collect: 
            [ :s | s = 'NA' ifTrue: ['NaN' asNumber] 
                            ifFalse: [s asNumber] ]

The collect method returns a new data series where each element has been transformed using the code block as it’s argument. In this can I’m checking for NA and converting to NaN or the real number. Similarily, dealing with a date axis involved manually transforming it into a julian day representation (so basically just a number) and back again for the label. Despite this I’ve come away with a pretty positive opinion of both these packages. The code in DataFrame is easy to approach and I’ve already tried to add in a small fix that improves the parsing of boolean types from CSV files. I also picked up a bug in the RSScatterPlot class that was fixed by the developers almost instantly. It’s definitely a small community but smalltalk makes it easy to investigate and patch bugs on the fly. It’s really quite cool and lends itself to agile development.

I’m going to keep using Pharo on the side. For the moment I don’t think it could replace python/R in my professional work but the language is easy and good for quicker analyses.

A Make for URIs

c.skennerton@gmail.com (Connor Skennerton) — Mon, 16 Nov 2020 18:07:45 -0800

Make has been a one of the key tools in my arsenal for gettings things done. Although it was developed for compiling code, its functionality can be generalized to any process that requires files to be generated based on dependancies. I recommend you look at these slides by Vince Buffalo as a good introduction to using make for scientific workflows. Make works by creating a dependancy graph of files and their prerequisites using the last time the file was modified as a way to determine if a file needs to be remade. This general concept is great for reproducable scientific research or many other repeating tasks and workflows. However it’s not without it’s flaws.

Make’s syntax is very obtuse using many shorthand variables like to describe rules that make it difficult to start using. But even after leaning its syntax I’ve continually found one fatal flaw: all of the files need to be local, on your current computer. This makes sense given its original function of compiling code. However in data analysis and scientific workflows we often have to interact with remote files on AWS S3 or files that we download from a web resourse. These files don’t have a time stamp that make can use and so their presense completely breaks the dependancy graph.

There are work arounds of course. I’ve used dummy empty files as a way of keeping track of the last time a URL was downloaded or just downloaded all of the external files at the beginning and end of an analysis, which is used in the popular cookiecutter data science template. But both of these solutions are brittle and don’t really solve the problem.

I looked for a URL enabled make, one that could integrate in remote files to the dependancy graph, but didn’t find anything suitable. So instead I set out to add this functionality to one myself.

I first looked at the source for GNU make however I couldn’t understand the source code so modifying that was out of the question. Instead I used mk as the base for my modifications. Mk was originally developed Plan 9 operating system as a re-write of make without many of its annoyances. I had stumbled upon a simple re-implementation of mk in Go by Daniel Jones a number of years ago and decided to add remote file support to that. The changes I describe below are available on my fork of mk on Github.

The goal of my improvemnts was to make the following rule work:

file.txt.gz: "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/001/696/305/GCF_001696305.1_UCN72.1/GCF_001696305.1_UCN72.1_feature_count.txt.gz"
    curl $prereq > $target

That is, that mk would recognise the prerequisite as a URL, determine if that URL was newer or older than the target and proceed accordingly. Since there is an extra colon in the URL we need to protect it with quotes so the mkfile parser doesn’t get confused.

The core of make and mk is deciding to remake a file based on whether its prerequisites are newer than it. It does this by looking at the last modified timestamp of a file. Sure enough inside graph.go there was a function updateTimestamp that gets the last modified time of a file or sets that the file doesn’t exist.

info, err := os.Stat(u.name)
if err == nil {
	u.t = info.ModTime()
	u.exists = true
	u.flags |= nodeFlagProbable
} else {
	_, ok := err.(*os.PathError)
	if ok {
		u.t = time.Unix(0, 0)
		u.exists = false
	} else {
		mkError(err.Error())
	}
}

This was the function to modify to look at time stamps of remote files. To do this we just need to identify files that look like remote files, i.e. start with http(s):// or s3://. The following simple modification makes that check and farms out the modification checking based on if it’s a http(s) or s3 remote file.

// u is a node in the dependancy graph.
// its name member is the full path of the file
if strings.HasPrefix(u.name, "s3://") || strings.HasPrefix(u.name, "https://") || strings.HasPrefix(u.name, "http://") {
up, err := url.Parse(u.name)
if err != nil {
	log.Fatal(err)
}

if up.Scheme == "http" || up.Scheme == "https" {
	updateHttpTimestamp(u)
} else if up.Scheme == "s3" {
	updateS3Timestamp(u, up)
}

The implementation of updateHttpTimestamp is pretty simple. A HEAD request is made to the URL and the Last-Modified header is read. If that header is present the time is parsed and used in the dependancy graph. If the header isn’t found it’s assumed that the URL doesn’t exist causing it to be remade.

func updateHttpTimestamp(u *node) {
	// get the headers of the URL
	resp, err := http.Head(u.name)
	if err != nil {
		log.Fatal(err)
	}
	lastModified := resp.Header.Get("Last-Modified")
	if lastModified == "" {
		// no Last-Modified header so lets assume that it
		// doesn't exist
		u.t = time.Unix(0, 0)
		u.exists = false
	} else {
		tmptime, err := time.Parse(time.RFC1123, lastModified)
		if err != nil {
			log.Fatal(err)
		}
		u.t = tmptime
		u.exists = true
	}
}

The implementation for updating an S3 file is similar but uses the AWS API to get the last modified time.

And with those small modifications the basic example I showed at the beginning now works. These modifications can be found in my fork of mk, try it out and see how much easier your make-ing becomes!

Drawing KEGG pathway maps using biopython and matplotlib

c.skennerton@gmail.com (Connor Skennerton) — Sun, 27 Sep 2020 22:11:47 -0700

I use KEGG a lot to understand microbial metabolism. KEGG is one of the largest resources of enzymes, biochemical reactions, genes, and molecules, all cross-linked and organized into what’s called metabolic maps. These maps are well-constructed images of enzymes that functions together for the same overall purpose like amino acid synthesis, or the metabolism of glucose. One of the great things about the website is the ability to color on your data to their metabolic maps. Doing this in bulk though can be very tedious as you need to view and download individual maps. Luckily, there is a great tutorial on how to dynamically color KEGG maps using biopython that I’ve used in the past to create PDF images.

While this works well, the current biopython graphics module is based on reportlab, which is a more niche graphics system than matplotlib. This means that the images made in biopython can’t easily be combined with other plots generated with matplotlib and the biopython implementation of drawing KEGG maps is restricted to producing PDF documents.

I was recently working on a project where I needed to combine the KEGG maps with other types of plots, generated via matplotlib and didn’t want to do it manually in a separate program. Instead I looked into converting the KEGG_vis module of biopython from reportlab to matplotlib. Thankfully, the code is quite short and self-contained, thus making the conversion process easy.

Visualizing KEGG data in biopython is driven by KGML, an XML markup which describes the placement of objects in a pathway map. Biopython parses this file and draws on the graphics elements based on this information. Converting the drawing code was pretty straightforward as many of the concepts between reportlab and matplotlib are the same. The bulk of the drawing code happens in the __add_graphics method, which is responsible for adding in the lines, circles, and rectangles described in KGML. There is almost a 1:1 mapping between the reportlab constructs and the equivalent matplotlib Patch API. For example drawing a line path in the original reportlab version looked like

p = self.drawing.beginPath()
x, y = graphics.coords[0]
p.moveTo(x, y)
for (x, y) in graphics.coords:
    p.lineTo(x, y)
self.drawing.drawPath(p)
self.drawing.setLineWidth(1)  # Return to default

Which starts a line at an xy-coordinate and then iterates through all of the remaining point in the path using the p.lineTo method. The translation to matplotlib results in very similar code

x, y = graphics.coords[0]
verts = [
(x, y),  # left, bottom
]

codes = [
	Path.MOVETO,
]
for (x, y) in graphics.coords:
	codes.append(Path.LINETO)
	verts.append((x,y))

path = Path(verts, codes)
patch = patches.PathPatch(path)    
self.ax.add_patch(patch)

The only major difference was that reportlab seems to use a global state to keep track of things like the line color and weight; throughout the original code there are calls to set the font, color, and line and then return to the original state after certain calls to draw a graphics object have been made. Alternatively, in matplotlib, these modifications are passed in directly when creating a new patch object. Internally I made this change to the API by adding a **kwargs argument to the __add_graphics method.

# The following code snippet demonstrates the change from reportlab
# which used calls like setStrokeColor to globally change the state
# and the new interface which passes these arguments into __add_graphics
# as kwargs, which will be applied to the matplotlib patches.
for ortholog in self.pathway.orthologs:
    for g in ortholog.graphics:
	#self.drawing.setStrokeColor(color_to_reportlab(g.fgcolor))
	#self.drawing.setFillColor(color_to_reportlab(g.bgcolor))
	self.__add_graphics(g, fc=g.bgcolor, ec=g.fgcolor)

This new version of the KEGG_vis library lives on my fork of biopython for now.

Checking back in on CRISPRs

c.skennerton@gmail.com (Connor Skennerton) — Mon, 07 Sep 2020 20:34:14 -0700

As part of my PhD thesis I studied an emerging field of bacterial adaptive immunity, known as CRISPR. At the time I was interested in tracking this type of immune system in bacterial communities to track co-evolution between bacteria and their viruses. For anyone interested and brave enough, here is a link to my thesis. After I finished up writing, submitting, and ultimately obtaining my PhD I realised that large portions of the literature review section would never be published in a scientific journal. I came to the realisation that if this summary of CRISPR (of which I was very proud of) would ever be read I needed it to be published in an “alternative” way.

That way was Wikipedia and one of my very first blog posts summarized my method for turning my literature review into a Wikipedia article. Perhaps my biggest motivation for contributing to the Wikipedia page was how out of date the page was. My literature review from my thesis had dozens of references and was many pages long yet much of this new information wasn’t in Wikipedia at all.

Towards the very end of my PhD the research community began studying CRISPR for gene editing which has resulted in amazing (and controversial) advances in science using this technique. While CRISPR research exploded I went off on a different research direction and eventually left academic research for a position in the biotechnology industry. Despite not being directly involved with CRISPR research anymore I still maintain two pieces of software, minced and crass, that are designed to detect CRISPR arrays in either genomes or metagenomes; and recently I got interested in how the CRISPR page on Wikipedia had changed since I made my contributions.

Not surprisingly the number of contributions has increased drastically from 2014 onwards as CRISPR gene editing got significantly more press coverage. But what did surprise me was Wikipedia’s inbuilt page statistics showing I’m the second highest author in terms of characters contributed. It’s been years since I contributed so my assumption was that most of what I wrote would have been edited away. Seeing my name so prominent really filled me with pride as this contribution is likely my most read piece of scientific literature I’ve ever made. Digging into the history of the article’s revisions it became clear that my name is still so high on the list due to the organic nature of Wikipedia article growth. I could see in the history section that large parts of the CRISPR page had been split off into the separate CRISPR gene editing page, which is itself a huge page; and the large cas9 page, which details the primary enzyme used in CRISPR gene editing. Intuitively this makes sense, people contribute small amounts to a page that already exists until someone decides that much of that information is better suited under a different title.

Some elements for my original contribution are still present including the table of signature CRISPR genes and a summary figure of the different types of CRISPR systems. Both of these elements are sorely out of date. I was disappointed to see that the table in particular hadn’t been updated as many of the rows still contained descriptions stating “function unknown”. Surely with the increased interest in CRISPR all of the proteins would have some more information about them.

Thankfully this assumption was true, much research on CRISPR has taken place but the information has yet to be disseminated back to Wikipedia. A quick literature search on Pubmed showed amazing research on many of the different cas genes. This let me change a few “function unknown” rows in the table and let me create a few new enzyme stubs on Wikipedia for cas2, cas3, and cas4. I hope that the additional pages will spurn either myself or others to add information. Right after I created the cas3 page another user added in some more information within a day!

Looking back at this has kindled my own interest in CRISPR again. A short summary of the information I found and integrated into Wikipedia while writing this blog post…

cas4

Is an endonuclease that works with cas1 and cas2 to generate spacer sequences. It looks for PAM sequences and processes the spacer before insertion into the CRISPR array.

cas5, cas6, cas7

These are essential for generating crispr-RNAs (crRNAs) which are used to target foreign DNA.

cas10

Binds to the 5’-end of the CRISPR RNA and promotes the oligomerization of other cas proteins creating the interference complex that destroys foreign DNA/RNA

Using Amazon Neptune full text search

c.skennerton@gmail.com (Connor Skennerton) — Sun, 23 Aug 2020 22:14:10 -0700

I’ve been trying out Amazon Neptune’s full text search feature. Overall it’s been a great experience although there are a few caveats when searching that means that you’ll have to craft your queries carefully to make full use of the feature.

The tinkerpop standard has some text searching features however it lacks any advanced features such as searching using regular expressions or even case-insensitive searching. It’s left to different implementations to augment this text searching capability.

The developers at Amazon Neptune chose to integrate Elasticsearch as their text searching engine, which offers a rich searching ability.

The integration between Elasticsearch and Neptune is pretty seamless with clear documentation. I used AWS’s managed elastic search service and I’m not sure if an external elasticsearch setup could work. I already had a Neptune database however to work with elastic search I needed to turn on the streams feature so that data could be replicated from one service to another. After turning on the streams feature I had to manually reboot my Neptune instances for it to take effect. With the streams feature turned on I used the export to elasticsearch cloudformation template to mirror the data into elastic search. This is a one time operation as long as the you also use the second cloudformation template that uses the Neptune stream to constantly update the elasticsearch index when changes are made to the neptune database.

Using the feature with gremlin also works great. When you want to run a query using full text search you need to add a withSideEffect step at the begining of the traversal

g.withSideEffect("Neptune#fts.endpoint", "<ENDPOINT_URL>")

And then after that you can use full text searching in a has step. The AWS docs have some good examples that show most of the features. Below are a couple of additional observations I made.

Full text searching is overloaded in the has-step. You by default you can search in a single property by using the form has("<property_key>", "Neptune#fts <query_string>".

The following searches for all nodes in the graph that have foobar in their name property.

g.withSideEffect("Neptune#fts.endpoint", "<ENDPOINT_URL>").
    V().
    has('name', 'Neptune#fts foobar').
    valueMap()

But you don’t have to specify a property. Using a * character in place of the property key allows you to search for the term in all of the properties of the nodes.

g.withSideEffect("Neptune#fts.endpoint", "<ENDPOINT_URL>").
    V().
    has('*', 'Neptune#fts foobar').
    valueMap()

Using this simple method is an all or one approach, you can’t restrict the search to multiple, known properties. To get this functionality you need to be a little more low-level and use the Lucene syntax, which the docs has some examples of near the end. To make use of this you’ll need to know a bit about how Neptune data is translated into elasticsearch documents and also read up on the query string syntax of elastic search.

Below the has-step makes use of a full text search looking at both the value of the name property and the other_name property. Due to the way Neptune data is translated into elasticsearch documents we need to refer to them as predicates.name.value and predicates.other_name.value

// note that there is an extra query hint to Neptune to use the query_string syntax for full text search
g.withSideEffect("Neptune#fts.endpoint", "<ENDPOINT_URL>").
    withSideEffect("Neptune#fts.queryType", "query_string").
    V().
    has('*', 'Neptune#fts predicates.name.value:foo* OR predicates.other_name.value:bar~').
    valueMap()

When looking at the query above you could try to formulate it in a more “gremlin” way by having the separate search terms inside a union-step, like below.

g.withSideEffect("Neptune#fts.endpoint", "<ENDPOINT_URL>").
    V().
    union(
        has('name', 'Neptune#fts foo*'),
        has('other_name', 'Neptune#fts bar~')
    ).
    valueMap()

However in my tests, I’ve found this to be very slow in comparison. Although I don’t know why that is we do have to remember that these queries are using two different engines and any full text search traversal has to go from Neptune through elasticsearch’s API and back again; so it’s probably best to make use of as much as the elasticsearch query language can offer so the two services have to interact the least amount.

An update on Bioawk

c.skennerton@gmail.com (Connor Skennerton) — Tue, 01 Oct 2019 17:26:23 +0000

Bioawk is a great project started by Heng Li some years ago. The aim was to take the awk source code and modify it slightly for use with common biological formats and adding in some new functions. Heng’s original doesn’t accept too many pull requests so to add in some features, I maintain my own fork that has a few improvements.

Long ago I added in a translate function and recently I added in a function that will take the attribute string from a GFF file and turn it into an awk array. It was possible to split the GFF attribute string in awk itself as the following awk code demonstrates

function gffattr(attr, array){
    # split the attribute string into key value pairs
    split(attr, a, ";")
    for (i in a) {
        # now split out the key and value
        split(a[i], b, "=")
        for (j in b) {
            array[j] = b[j]
        }
    }
}

but this is a bit clunky and not suited for the kind of one-liner processing that awk excels at. So now with the latest update of my bioawk fork this function is built-in. If you want to filter a GFF on some attribute you can now do something like the following

bioawk -c gff '{gffattr($attribute, a); if(a["some_attr"] == "some_value") print}'

Other random updates

The base awk source code has been getting a lot more love recently and my fork has so far merged in all of the upstream updates so there are a few new bug fixes in there.

Bioawk used to ignore the output field separator when printing the whole record – this is no longer the case in my fork.

Getting data from NCBI assembly using the accession number only

c.skennerton@gmail.com (Connor Skennerton) — Tue, 10 Apr 2018 23:18:11 +0000

NCBI’s assembly database is a great one-stop-shop for genomic data and annotations but it’s actually kind of difficult to download data if you only know the accession number of an assembly. The documentation says that the assembly database is integrated with entrez-direct, a great set of command line utilities for accessing NCBI data from the command line. Most of the databases have an option to download data based on the ID, so I thought that something like the following would work

efetch -id GCA_000398025.1 -db assembly -format fasta

Alas it does not, however the directory structure for the assembly FTP site nicely mirrors the accession numbers so for the example above, the path looks something like this: ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/398/025/ Below is a little awk script that I wrote to generate the FTP path and for input into wget. The input for this script is a file containing the accession in the first column and optionally a second column containing the assembly name. The full FTP path contains both of these but we can get around not having the assembly name by using wildcards. Nope, you need both the accession and the assembly name for this snippet to work. But I wrote a more expansive script in python that do it with just the accession number, you should probably use it instead.

BEGIN{OFS="/"}
{
  accession = $1;
  joined_acc = accession
  if (NF == 2) {
    joined_acc = joined_acc"_"$2;
  } else {
    joined_acc = joined_acc"*";
  }
  split($1, a, "_");
  sub(/\\.[[:digit:]]$/, "", a[2]);
  gsub(/.{3}/,"&/", a[2]);
  print accession"\tftp://ftp.ncbi.nlm.nih.gov/genomes/all",a[1],a[2],joined_acc,joined_acc"_genomic.fna.gz"
}

Creating huge metabolic overviews for comparative genomics

c.skennerton@gmail.com (Connor Skennerton) — Fri, 15 Dec 2017 19:38:46 +0000

I love looking at KEGG maps and using them to understand an organisms metabolism but they have their limitations. For starters, you’re obviously stuck with how they are drawn, which in most cases includes many variations on a particular pathway. Secondly, the tools for mapping on your own genes to a pathway are limited to one organism at a time. What I really wanted was a way of quickly comparing 10s of genomes across a common set of pathways, in this case amino acid synthesis pathways. Failing to find a suitable way to map on many genomes to KEGGs pathway images, I decided to roll my own. I mixed up manually drawing the pathways and then using code to color in the presence/absence of the reaction in each organism. My solution starts by creating a template svg image in Illustrator. This is the laborious part as I went through and redrew the amino acid pathways using information from both KEGG and MetaCyc. For each enzyme I positioned a grid of boxes, where each box represents an individual genome, that I would eventually color in based on the presence/absence of that reaction in that organism.

The TCA cycle and parts of the amino acid synthesis pathways. Each reaction is labelled with the EC number and contains a grid of 100 boxes.

Now the trick is to group the parts of the enzyme together and give that group a name in Illustrator. First the grid of 100 boxes was grouped first per row and then I grouped those 10 rows together (this is important and I’ll explain why below). And then the boxes, the arrow and the text label all got grouped together. Next name both the larger group and the boxes in Illustrator. You can do this easily by double clicking the group to enter “isolation mode”, where everything that isn’t in the group gets faded, and then changing the name in the layers panel of Illustrator.

When in isolation mode, the layers panel gives easy access to changing the name of the group. In this case I’m calling the group after the EC number.

Now comes the critical part in naming - the group containing only the 100 boxes must be start with “box-” and then what follows will be the identifier that gets used for coloring the boxes.

each enzyme has a sub-group containing the matrix of 100 boxes. This group must start with ‘box-’ and then whatever follows will be used to lookup that reaction for each organism.

In the case above I’m using the EC number again as my identifier. It is whatever is written after the “box-” in the name of the group that is important. This means that you could use any kind of label in the figure, for simplicity I’ve simply kept both the visual label of the enzyme and the name of the group to both use the EC number. Now, the reason why this is important is that we are going to save our figure as an svg file. Svg is an xml based file format, which means that it is simply text and thus is easy to modify using some scripting. Lets have a look at how each enzyme gets encoded in svg:

svg of part of one enzyme. the tag in svg is a group. You can see that the names for the whole enzyme and for the group of 100 boxes are retained from Illustrator

The names that were given to the groups in Illustrator have been retained in the svg file as the id tags, and it is this text that we’ll process using the script. The other thing to note is how the boxes are arranged, in this case as groups of 10 - these are the rows that were made originally.

Coloring the boxes programmatically

Now for the scripted part. You’ll need the code available here, which will color in the boxes based on a couple of text files. Run the script as follows: python color_template.py -e ec_numbers.csv -i template.svg -o colored.svg -c organism_colors.csv The ec_numbers.csv file must contain three columns and must have a header row labelled organism, ec, lineage. The organism column is the name of the organism, the ec column must contain the identifiers that match up to the names used in the svg file. For example, in the svg file, if there is a box called “box-1.1.1.1” then the corresponding annotation in the ec column should be 1.1.1.1. However, there isn’t a requirement that this column be EC numbers or even that they are all from the same source. You could combine EC numbers, KO numbers and Pfam families in this column they just have to match what’s written in the svg file. The lineage column should contain information on how to properly order all of the genomes into the 100 boxes. I use the full taxonomy string of the organism to allow for similar organisms to be grouped together. However, this column doesn’t require a taxonomy string and is simply used for lexicographic sorting, so you could put anything in there to sort organisms however you want. The organism_colors.csv file should contain two columns with a header row labelled organism and color. The organism column is the organism name that was used in the ec_numbers.csv file. The color column should contain a valid css color.

Formatting scientific names in R

c.skennerton@gmail.com (Connor Skennerton) — Sat, 22 Jul 2017 17:27:02 +0000

Here is a function that will take a character string in R and return an expression for fancy formatting in plots that properly italicize scientific names. The syntax for doing this is truly quite horrible, but this is how R does it.

scientific_name_formatter <- function(raw_name) { 
    # strsplit returns a list but we are passing in only 
    # one name so we just take the first element of the list 
    words = strsplit(raw_name, ' ', fixed = TRUE)[[1]] 
    # some sort of acronym or bin name, leave it alone 
    if(length(words) < 2) { return(raw_name) } 
    else if (length(words) > 2) { 
        if (words[1] == 'Candidatus') { 
            # for candidatus names, only the Candidatus part is italicised # name shortening it for brevity 
            unitalic <- paste(words[2:length(words)], collapse = ' ') return(bquote(paste(italic(Ca.) ~ .(unitalic)))) } 
        else if (grepl('^\[A-Z\]+$', words[1])) { 
            # If the first word is in all caps then it is an abreviation 
            # so we don't want to italicize that at all 
            return(raw_name) } 
        else { 
            # assume that everything after the second word is strain name # which should not get italicised 
            unitalic <- paste(words[3:length(words)], collapse = ' ') return(bquote(paste(italic(.(words[1])) ~ italic(.(words[2])) ~ .(unitalic) ))) 
        } 
    } else { 
        return(bquote(paste(italic(.(words[1])) ~ italic(.(words[2]))))) 
    } 
}

I use it like the following in ggplot by setting the labels in scale_y_discrete.

scale_y_discrete(limits = df$organism_name, labels = sapply(df$organism_name, scientific_name_formatter ))

split_by: splitting files on the command line based on content

c.skennerton@gmail.com (Connor Skennerton) — Sun, 09 Apr 2017 08:04:41 +0000

Unix has so many great ways to perform text manipulation but one niche which hasn’t been filled is splitting a tabular file into pieces based on the contents of certain columns. There are two commands, split and csplit, that do a similar role. split can split a file into a certain number of bytes or lines; csplit uses a regular expression to determine where to split the file. Often for my purposes neither of these tools is a good fit, and what I really want is an equivalent to the “group by” clause in SQL databases. Group by sorts rows based on certain grouping columns so that you can then perform summaries on that group. split_by splits out a file based on the grouping columns for further processing on each chunk. For example, say I’ve got a delimited text file containing a mix of categorical and numerical data like the following:

A	B	C	D	E	F
AU	BR	447	223.2	55958	US
AU	FR	348	16.8	58484	AU
US	UK	12	53.30	129	PG

I want to split this file into smaller files based on what’s in column A, which in this example would make two files for the 2 lines containing “AU” and for the 1 line containing “US”. Neither, split or csplit really handle this case and the only option is to use an awk, perl or other script to handle it. (It’s also possible to do this in multiple passes with grep) After writing similar awk one-liners to do this sort of thing I decided to make it a bit more reusable:

#!/usr/bin/awk -f
BEGIN{
    split(cols, group_by, ",")
    col_sep = "_"
}
{
    if(header == 1 && NR == 1) {
        next
    }
    out_file = ""
    for(col in group_by) {
        if(col == 1) {
            out_file = group_by[col]
        } else {
            out_file = out_file col_sep group_by[col]
        }
    }
    out_file = (prefix out_file suffix)
    print $0 > out_file
}

That’s the whole program! You can run it like the following, taking advantage of the variables defined in the script:

./split_by -vFS="\t" -vprefix=test_split_ -vsuffix=".tsv" -vcols=1 -vheader=1 file.tsv

This will produce two files on the example above called “test_split_AU.tsv” and “test_split_US.tsv”. Only the cols argument is needed to say which of the columns is used to split on. If you want to split on multiple columns pass a comma separated list -vcols=1,2. The output file names are generated based on the contents of those columns so if there are any special characters in there that might mess up a file name, you’re in trouble. The prefix and suffix variables default to nothing but are useful for being able to find the files for later by giving them all a common prefix. The nice thing about this is that all of the awk variables are still available so changing between a csv and tsv file can be achieved using the FS parameter on the command line.

Downloading NCBI genomes from a given taxonomy

c.skennerton@gmail.com (Connor Skennerton) — Sun, 20 Nov 2016 05:18:43 +0000

The Entrez Direct toolkit is great for programmatic access to all of NCBI’s resources. This little snippet below finds all of the refseq representative genomes for a given NCBI taxonomy ID, makes a little summary of the genomes downloaded and uses wget to download the genbank files from the Assembly FTP. Change the inital query in the first call to esearch to change what genomes are downloaded.

esearch -db assembly -query "txid28890[Organism:exp] AND representative [PROP]" |
 efetch -format docsum |
 xtract -pattern DocumentSummary -element 
    AssemblyAccession SpeciesTaxid SpeciesName FtpPath_RefSeq |
 sed 's/,.\*//' |
 sort -k 3,3 |
 tee downloaded_genomes.tsv |
 cut -f 4 |
 sed -e 's/$/\\/\*genomic.gbff.gz/' |
 wget -i /dev/stdin

Python ete3: formatting organism names the way I want

c.skennerton@gmail.com (Connor Skennerton) — Thu, 17 Nov 2016 00:58:40 +0000

I feel like I’m on a life-long quest to make all of my phylogenetic tree figures completely programmatically. The best tool I’ve found for making them is the ete library for the python programing language. I’ve already figured out how to get trees drawn in the style that I like but there was still one thing left to do: making organism names italicize correctly. I work with microorganisms where the convention is for the genus and species names to be italicized but the strain name to be in regular font. For example: Methanoregula formicica SMSP, DSM 22288. There are exceptions to this as well; if an organism is not in pure culture then the name Candidatus is prepended and any genus and species names remain in regular format. Unfortunately, ete doesn’t have an interface for doing this kind of name formatting, you can either have it italic, bold or regular, but you can’t mix and match. Happily though it does contain an interface for dropping down into the lower-level drawing engine (pyQt4), which enables you to do all sorts of custom things. Below is my solution to making fancy formatted organism names. I think it might be a little hacky but seems to work. The organism name should be in the “name” attribute of the leaf for this to work. (See the example below) Here are the formatting rules:

If the name has less than 2 words: do nothing. This is to catch internal names or accession numbers that are not valid scientific names
If the first word of the name is “Candidatus”: abbreviate it to “Ca.” and italicize just that part.
If there are more than 2 words: italicize the first two.
If there are exactly 2 words: italicize both

from ete3 import Tree, faces, TreeStyle
from PyQt4 import QtCore
from PyQt4.QtGui import QGraphicsRectItem, QGraphicsSimpleTextItem, \
QGraphicsPolygonItem, QGraphicsTextItem, QPolygonF, \
     QColor, QPen, QBrush, QFont

def scientific_name_face(node, *args, **kwargs):
    scientific_name_text = QGraphicsTextItem()
    words = node.name.split()
    text = []
    if len(words) &lt; 2:
        # some sort of acronym or bin name, leave it alone
        text = words
    elif len(words) &gt; 2:
        if words[0] == 'Candidatus':
            # for candidatus names, only the Candidatus
            # part is italicised
            # name shortening it for brevity
            text.append('<i>Ca.</i>')
            text.extend(words[1:])
        else:
            # assume that everything after the
            # second word is strain name
            # which should not get italicized
            text.extend(['<i>'+words[0],words[1]+'</i>'])
            text.extend(words[2:])
    else:
        text.extend(['<i>'+words[0],words[1]+'</i>'])

    scientific_name_text.setHtml(' '.join(text))

    # below is a bit of a hack - I've found that the height of the bounding
    # box gives a bit too much padding around the name, so I just minus 10
    # from the height and recenter it. Don't know whether this is a generally
    # applicable number to use
    masterItem = QGraphicsRectItem(0, 0,
        scientific_name_text.boundingRect().width(),
        scientific_name_text.boundingRect().height() - 10)

    scientific_name_text.setParentItem(masterItem)
    center = masterItem.boundingRect().center()
    scientific_name_text.setPos(masterItem.boundingRect().x(),
        center.y() - scientific_name_text.boundingRect().height()/2)

    # I don't want a border around the masterItem
    masterItem.setPen(QPen(QtCore.Qt.NoPen))

    return masterItem

def master_layout(node):
    if node.is_leaf():
        F = faces.DynamicItemFace(scientific_name_face)
        faces.add_face_to_node(F, node, 0)

if __name__ == '__main__':
    t = Tree()
    # this is fake data to show the rendering
    t.populate(4)
    leaves = t.get_leaves()
    # give the leaves some different types of names
    leaves[0].name = 'Methanosarcina barkeri'
    leaves[1].name = 'Candidatus Methanoperedens nitroreducens'
    leaves[2].name = 'ANME_bin_23'
    leaves[3].name = 'Methanosaeta thermophila PT'

    ts = TreeStyle()
    ts.show_leaf_name = False
    ts.layout_fn = master_layout
    t.show(tree_style=ts)

My guide to annotating proteins and pathways

c.skennerton@gmail.com (Connor Skennerton) — Thu, 22 Sep 2016 22:14:50 +0000

So. You’ve got yourself a nice new genome sequence and you want to know what kind of metabolism it has. There is a good chance that you have some idea already — you think it’s a nitrogen fixer or a sulfate reducer etc. — based on other analyses and now it’s time to strengthen your paper with a bit of genomic evidence.

Getting an initial annotation

The vast majority of the genes in the genome are going to be hypothetical proteins; of the rest, a sizable chunk are going to be genes with a general sort of annotation like “ABC transporter” (which says nothing about what it’s transporting), and the rest are going to be metabolic genes that you probably care about. The first thing that you want to do is to throw your genome into one of the automatic annotation pipelines (prokka, rast, img, ncbi, kaas) to get an initial annotation. You can then search through these to look for pathways of interest. However, don’t just straight out believe what these pipelines say, they will definitely miss some genes and report annotations that aren’t correct, but they are a good start since in most cases you will only be interested in a portion of the genes. The most important thing to remember is to be a scientist. Have a sceptical mindset and interrogate your data. You can’t look at every gene (well you can but probably don’t have time) so my rule of thumb is that you should manually check all of the genes in all of the pathways that you are going to write about in your paper.

My Workflow

1. Compare the sequence of your unknown gene against Uniprot.

You can also use NCBI blast database or IMG or any other database you wish however I like Uniprot because it ranks it annotations with an overall confidence score, so you know what kind of experiments has informed the annotation. Uniprot will show some starred proteins that are biochemically verified - this is a good way to figure out if your gene is annotated with the correct function.

Remember that most genomes are automatically annotated which means that even if all of the top blast hits are annotated as the same thing, it doesn’t mean that it’s correct. Comparing it against genes that have been biochemically annotated is important.

Also remember that blast is a local aligner, so don’t just look at the e-value, look at the percentage matched of the query (your gene) and the subject (gene in the database). If you have a very long gene and only part of it matches then you should definitely investigate this!

2. Find conserved domains using Interpro

Sometimes (read: Often) blast matches will not be conclusive or return only a general annotation. In these situations it’s best to look at the protein sequence another way. In this case looking at the domains that make up the protein using Interpro may help figure out what it does. This is also a good way to compare your protein to a biochemically analysed protein. I your gene has extra domains, different domains, or fewer domains then it may well be that the blast annotation was incorrect.

3. Look at surrounding genes

A lot of enzymes are encoded by multiple genes which all exist in one operon. If you want to say that your bug has some enzyme you should make sure that it contains all of the subunits for that enzyme. This can be an essential step where enzymes evolved from a common ancestor such that individual subunits can be difficult to distinguish. A good example here is NADH dehydrogenase and the evolutionarily related hydrogenases, which can be easily distinguished by the operon structure, but individual subunits often get mis-annotated.

4. Make an alignment (optional)

The last thing you might need to do is to make an alignment to see if your putative enzyme has all (or some of) of the right conserved residues as other biochemically characterised proteins. If the alignment looks terrible then perhaps you’ve got something novel.

Messing around with Phyloxml in ete3

c.skennerton@gmail.com (Connor Skennerton) — Fri, 12 Aug 2016 22:09:04 +0000

ete3 has support for phyloxml which I use with archaeopteryx tree viewer for a lot of my day-to-day phylogenetics visualisation. My main reason for using phyloxml is one of convenience as I have a script that will easily add in the proper organism name onto the tree and I think that archaeopteryx is a really good basic tree viewer. I wanted to draw a tree from phyloxml in ete using my own style and to have the proper organism name to be rendered. In my phyloxml file I have this coded in as the scientific name for each leaf (see below for phyloxml snippet), so now all I needed to do was make this the node name when rendering the tree.

<clade> 
  <name>IMG_2526164742</name> 
  <branch_length>0.19955</branch_length> 
  <taxonomy> 
    <scientific_name>Desulfobacterium anilini DSM 4660</scientific_name> 
  </taxonomy>
</clade>

Easy, right? Wrong. I found that the interface for phyloxml was not the same as for newick formatted trees and unfortunately the documentation for phyloxml in ete3 is a bit lacking as there wasn’t a complete listing of methods for each class. After much messing around, looking at the source code of ete3 and examining python objects using the builtin dir function I was able to get what I wanted. turns out that for each node/leaf I needed to access the phyloxml_clade attribute, which has an attribute taxonomy, which implements an iterable interface (I think it’s probably a list), which I could then use to access the scientific name and make the name of the leaf for printing. It’s a little convoluted but easy when you know how.

from ete3 import Phyloxml
project = Phyloxml()

# iterate through the trees in the phyloxml file
for tree in project.get_phylogeny():
    # go through the node in the tree
    for node in tree:
        # assign the node name from the data in the phyloxml file
        node.name = node.phyloxml_clade.taxonomy[0].get_scientific_name()

tree.show()

Going from a messy supplementary table to good clean data

c.skennerton@gmail.com (Connor Skennerton) — Mon, 11 Jul 2016 21:00:53 +0000

Bioinformatics… Or ‘advanced file copying’ as I like to call it.

— Nick Loman (@pathogenomenick) January 29, 2014

Get ready for some advanced file copying! I recently had to clean up some data from the supplementary material from Pereira et. al 2011, which is a very nice table of manually annotated genes in sulfate reducing bacteria. The only problem is that the table is designed for maximum human readability, which made it a real pain when trying to parse out the data. I decided to use R and the “tidyverse” packages to clean up the table to make things work better for downstream analyses. This isn’t part of my normal workflow, I’m more of a python guy, but after doing this analysis in R I’d have to say that I’m a convert.

Getting the data

Download the supplementary material from the paper linked above, if you want to play along at home. This data is a nicely formatted excel workbook containing eight spreadsheets with the locus identifiers for a number of genes important in suflate reducing prokaryotes. While this data is nice and visually appealing, it is not tidy and it’s difficult to get the information I want out of it. I want the locus identifiers for the genes of interest so that I can download them from NCBI and use them as a blast database.

Cleaning the data

Some things are just easier to do in excel before tidying the data in R here is what I did:

removed the empty columns and rows at the beginning. This actually isn’t difficult to do in R, but doing this makes inputting the data more painless cause then R will pick up the column names.
Remove the rows that contain just the taxonomic information
For some of the sheets (‘Hases’, for example) I removed rows at the beginning that gave hierarchy to the columns. These are mostly unnecessary and make it difficult to parse the excel sheet, as readxl does not currently handle merged cells and cause the boundaries of this hierarchy is coded visually using cell boarders in excel.
For some reason there were single quotation marks in the Archaeoglobus fulgidus DsrK locus identifiers, which I removed

Open up an R session and load the following libraries (assuming that you already have them installed)

library(tidyr)
library(readxl)
library(stringr)
library(dplyr)

Import the data into R using readxl. Creates a list of dataframes.

d <- lapply(excel_sheets("~/Downloads/data_sheet_2.xls"), \
read_excel, path = "~/Downloads/data_sheet_2.xls")

remove the completely empty rows

d <- lapply(d,
function(n) n[rowSums(is.na(n)) != ncol(n),])

Lets look at what our table looks like (Note the ‘organism’ column is not shown for brevity)

knitr::kable(d[[1]][,-1])

locus	SAT	AprA	QmoA	DsrA	DsrC	H-Ppi	FdxI	FdxII
AF	1667	1670	0663	0423	2228	NA	00427; 1010; 0355; 0923; 2142; 0166; 1700; 0156; 0464	0167
Arcpr_	1264	1261	1260	0139	1726	NA	0142; 0625; 0483; 0712; 1058	NA
Cmaq_	0274	0273	NA	0853	0856	0949	0549; 1711	NA
DaesDRAFT_	2031	2029	2028	2438	0796	NA	1729	0903
Dde_	2265	1110	1111	0526	0762	NA	3775	0286
Ddes_	0454	2129	2127	2275	1917	NA	889	NA
DMR_	39470	05400	05410	03600	15890	NA	39570; 18760	13970
DESPIG_	02241	02773	02771	NA	02353	NA	00991	NA
Desal_	0228	0230	0231	0787	0984	NA	1299	0241; 2850
DFW101DRAFT_	0832	1162	1163	3451	2958	NA	0847	0729
DVU	1295	0847	0848	0402	2776	NA	3276	NA
Dbac_	3196	3198	3199	0279	2958	NA	0275	2977
Dalk_	2445	1569	1568	4301	4140	3373	4380; 2230; 2714	2374
HRM2_	31180	04510	04500	42400	22050	NA	26720; 10680; 01580; 39570	40690
Dole_	1002	0998	0999	2669	0463	2820	1168	2655
Dret_	1968	1966	1965	0244	1739	NA	0240	0154; 0169
DthioDRAFT_	1410	1407	1406	2272	2675	NA	2268	NA
DP	1472	1105	1106	0797	0997	NA	2755; 0775	1865
DaAHT2_	0293	1471	1470	0296	2041	NA	1668	2532; 2287
Sfum_	1046	1048	1287	4042	4045	3037	4046	2933; 3217
Dtox_	3579	3577	3576	0079	0077	3931	0074; 0532; 1221; 1608; 3208	1637
Dred_	0635	0637	0638	3187	3197	2985	3200; 2937; 0466	2203
Daud_	1076	NA	1884	2201	2190	0308	2193; 1963	1080
Adeg_	1712	1080	1079	2094	0035	NA	0032	1625
THEYE_	A1835	A1832	A1831	A1994	A0003	NA	A0789	NA

In the data, the columns for each gene are really values, not variables; they should be converted into individual rows. To do this use the gather function from tidyr. Here I specify the name of the new columns gene.identifier which will contain the name of the gene and locus.identifier which will contain the information for that gene. I’m also setting na.rm which will not include genes where it was not found in the organism. After the gather function is applied all of the data frames in the list will have the same columns, which means that they can all be concatenated into one big data frame. To do this I’m using dpylr::bind_rows.

d <- lapply(d, function(n){ n %>% 
gather(gene.identifier, locus.identifier, 
-c(organism, locus), na.rm=TRUE)}) 
d <- bind_rows(d)
## Warning in rbind_all(x, .id): Unequal factor levels: coercing to character
knitr::kable(d[130:140,])

organism	locus	gene.identifier	locus.identifier
Archaeoglobus fulgidus	AF	FdxI	00427; 1010; 0355; 0923; 2142; 0166; 1700; 0156; 0464
Archaeoglobus profundus	Arcpr_	FdxI	0142; 0625; 0483; 0712; 1058
Caldivirga maquilingensis	Cmaq_	FdxI	0549; 1711
Desulfovibrio aespoeensis	DaesDRAFT_	FdxI	1729
Desulfovibrio desulfuricans G20	Dde_	FdxI	3775
Desulfovibrio desulfuricans ATCC 27774	Ddes_	FdxI	889
Desulfovibrio magneticus RS-1	DMR_	FdxI	39570; 18760
Desulfovibrio piger	DESPIG_	FdxI	00991
Desulfovibrio salexigens	Desal_	FdxI	1299
Desulfovibrio sp. FW1012B	DFW101DRAFT_	FdxI	0847
Desulfovibrio vulgaris Hildenborough	DVU	FdxI	3276

The other untidy aspect of the dataset is that there are multiple locus identifiers for some of the genes (presumably cause there are multiple copies in the genome). We next need to split them out into separate observations (rows). The str_split function from stringr will split a string based on a regular expression and return a list of values. I then pass this to the unnest function, which will “flatten” the list into multiple rows as required.

d %>% mutate(locus.identifier = 
str_split(as.character(locus.identifier), "; |\\/")) %>% 
unnest(locus.identifier) -> d
knitr::kable(d[130:140,])

organism	locus	gene.identifier	locus.identifier
Archaeoglobus fulgidus	AF	FdxI	00427
Archaeoglobus fulgidus	AF	FdxI	1010
Archaeoglobus fulgidus	AF	FdxI	0355
Archaeoglobus fulgidus	AF	FdxI	0923
Archaeoglobus fulgidus	AF	FdxI	2142
Archaeoglobus fulgidus	AF	FdxI	0166
Archaeoglobus fulgidus	AF	FdxI	1700
Archaeoglobus fulgidus	AF	FdxI	0156
Archaeoglobus fulgidus	AF	FdxI	0464
Archaeoglobus profundus	Arcpr_	FdxI	0142
Archaeoglobus profundus	Arcpr_	FdxI	0625

Now for the final clean-up. The original data separated the locus prefix and the locus identifier, now I want to combine them back together. To do this I’m going to use a couple of calls to the mutate function, which modifies a column. First, in the previous command I converted the locus.identifier column to characters, however this has the unwanted effect of having decimal places in the strings, which I don’t want. Passing the locus.identifier column to the sub function will remove the unwanted text. The next mutate command combines the locus and locus.identifier columns into one and finally I select which columns I want in the final data frame using the select function.

d %>% mutate(locus.identifier = 
sub("\\.0+","",locus.identifier, perl=T)) %>% 
mutate(locus = paste0(locus,locus.identifier)) %>% 
select(organism, locus, gene.identifier) -> d
knitr::kable(d[1:10,])

organism	locus	gene.identifier
Archaeoglobus fulgidus	AF1667	SAT
Archaeoglobus profundus	Arcpr_1264	SAT
Caldivirga maquilingensis	Cmaq_0274	SAT
Desulfovibrio aespoeensis	DaesDRAFT_2031	SAT
Desulfovibrio desulfuricans G20	Dde_2265	SAT
Desulfovibrio desulfuricans ATCC 27774	Ddes_0454	SAT
Desulfovibrio magneticus RS-1	DMR_39470	SAT
Desulfovibrio piger	DESPIG_02241	SAT
Desulfovibrio salexigens	Desal_0228	SAT
Desulfovibrio sp. FW1012B	DFW101DRAFT_0832	SAT

Drawing Phylogenetic Trees, Connor Style

c.skennerton@gmail.com (Connor Skennerton) — Sat, 09 Jul 2016 00:23:54 +0000

I do a lot of work in phylogenetics, which means that for just about every paper I’ve written I’ve had at least one figure that is a phylogenetic tree. Making pretty looking trees for a publication is tedious and my previous workflow involved using ARB for actually drawing the tree and producing an initial file in postscript, and then loading that into Adobe Illustrator to make everything beautiful. The problem with this is that it is not an automated process so any time I need to change the tree I need to redo all of the ‘beautifying’ manually. Recently I coded up an alternative approach using the excellent ete package for python to draw trees exactly how I want. One of the nicest things about drawing trees in ARB is that you can collapse clades into wedges. Unfortunately, while ete does allow you to collapse clades it doesn’t provide a way to show the collapsed node as a wedge, the only options are a square or a circle. But there is the option to create a custom face which is exactly what I did. Below is a function to create ARB-style wedges:

 def polygon_name_face(node, width, height, width_percent): 
    """create a wedge shaped face in the style of ARB 
    
        Args: 
            width (int): size in pixels for the width of the wedge 
            height (int): size in pixels for the height of the wedge 
            width_percent (float): change the angle of the point of the wedge. This must be a number between 0 and 1
        
        Returns: QGraphicsRectItem: The Qt graphics item of the polygon 
    """ 
    points = [ (0.0, 0.0), # top left point 
               (width, 0.0), # top right point 
               (width * width_percent, height), # bottom right point 
               (0.0, height), # bottom left point 
               (0.0, 0.0) # back to the beginning 
             ]
    shape = QPolygonF() 
    for i in points: 
        shape << QtCore.QPointF(*i) 
        ## Creates a main master Item that will contain all other elements 
        ## Items can be standard QGraphicsItem 
        masterItem = QGraphicsRectItem(0, 0, width, height) 
        # Keep a link within the item to access node info 
        masterItem.node = node 
        # I dont want a border around the masterItem 
        masterItem.setPen(QPen(QtCore.Qt.NoPen)) 
        polygon = QGraphicsPolygonItem(shape, masterItem) 
        # Make the wedge grey in color 
        polygon.setBrush(QBrush(QColor( '#D3D3D3'))) 
        # Print the name of the node 
        text = QGraphicsSimpleTextItem(node.name) 
        text.setParentItem(polygon) 
        # Center text according to masterItem size 
        tw = text.boundingRect().width() 
        th = text.boundingRect().height() 
        center = masterItem.boundingRect().center() 
        text.setPos(center.x() + tw/2, center.y() - th/2) 
        polygon.setPos(0, masterItem.boundingRect().y()/1.5) 
        return masterItem

And then to actually use it in a script, set up the tree style. I like to mark internal nodes with bootstrap support >70% with a grey circle and >90% with a black circle as well. Below is the function that I use to add in the groups.

 def master_ly(node): 
    style = NodeStyle() 
    style['shape'] = 'circle' 
    if node.support >= .90: 
        style['size'] = 5 
        style['fgcolor'] = 'black' 
    elif node.support >= .70: 
        style['size'] = 5 
        style['fgcolor'] = 'grey' 
    else: 
        style['size'] = 0 
    
    if node in grouping_nodes: 
        style['draw_descendants'] = False 
        # Create an ItemFAce. First argument must be the pointer to 
        # the constructor function that returns a QGraphicsItem. It 
        # will be used to draw the Face. Next arguments are arbitrary, 
        # and they will be forwarded to the constructor Face function. 
        # in this case we pass through the width, height, and width_percent for 
        # the wedge. 
        F = faces.DynamicItemFace(polygon_name_face, 60, 30, 0.25) 
        faces.add_face_to_node(F, node, 0) 
        node.set_style(style)

Finally putting it all together

from ete3 import Tree, faces, NodeStyle, TreeStyle 
# We will need to create Qt4 items for making our custom polygon 
from PyQt4 import QtCore from PyQt4.QtGui 
import QGraphicsRectItem, QGraphicsSimpleTextItem, \ 
       QGraphicsPolygonItem, QPolygonF, QColor, QPen, QBrush 
# Populate this list with the root node of a clade 
# that should be turned into a wedge 
grouping_nodes = [] 
# load in your tree from somewhere, this is for fake data 
t = Tree() t.populate(30) ancestor = t.get_common_ancestor("aaaaaaaaa", "aaaaaaaaac") 
grouping_nodes.append(ancestor) 
ts = TreeStyle() 
ts.layout_fn = master_ly 
# order the subtrees in ascending order 
t.ladderize(1) 
t.show(tree_style=ts)

There are improvements to be made with the way I’m drawing the wedge. First, there isn’t any border between the top of the wedge and the next leaf — you can see the name “aaaaaaaaad” is a bit cramped. Second, ARB has a nice feature which changes the wedge dimensions based on the number of grouped leaves which I haven’t yet implemented.

Genome Bin Decontamination

c.skennerton@gmail.com (Connor Skennerton) — Sat, 03 Oct 2015 00:00:00 +0000

Genome bins comming off automated pipelines can be contaminated with parts of other genomes. As part of my workflow I use CheckM (I’m biased since I’m a coauthor) to assess the contamination of genome bins using single-copy marker genes. If you’re lucky then the genome bins that you’re interested in will be relatively complete without much contamination. Unfortunately that isn’t always the case. In this blog post I’m going to run through some of the analyses that I did on a genome bin that was 90% complete but 70% contaminated. This is exploratory analysis to see if I can manually improve the bin over what the automated tools can do.

Background

I’m not going into details on how I got this bin but briefly I had three metagenomic samples, named 3730, 5133, and 5579, which were assembled with megahit and binned using metabat. I ran the genome bins from sample 3730 through CheckM. In the below analysis I’m going to try to improve bin_41.

Analysis

Creating the input files

For starters I want the .depth.txt file created by metabat, which I copied and renamed 3730_coverage.tsv. Next I created a mapping file for which contigs belonged to which bins. I have the fasta files of all of the bins in a separate directory. To get the mapping of the fasta files I ran the following:

grep -oP '(?<=^\>)\S+' *fa | awk 'BEGIN{FS=":";OFS="\t"}{print $2,$1}' > 3730_bin_mapping.tsv

And finally I wanted to know where all of the multi-copy markers were so I created a file based on the CheckM output, with a bit of reformatting in awk:

checkm qa -o 6 ckm_results/lineage.ms ckm_results | awk 'BEGIN{FS="\t";OFS=","}{n = split($3,a,",");for(i = 1; i <= n; ++i){print $1,$2,a[i]}}' > 3730_multiple_markers.csv

Exploratory analysis in R

library(readr)
duplicated_markers = read_csv("3730_multiple_markers.csv")
coverage = read_tsv("3730_coverage.tsv")
# no header in the file, so give the columns names
bin_mapping = read_tsv("3730_bin_mapping.tsv", col_names = c("contigName", "bin"))

Clean up the dataframes to make the column names consistent and remove a bit of unneeded text. Note that these commands are specific to how files were named on my system, you may not need to do this or change this section to meet your own needs.

bin_mapping$bin = gsub("metabat_binned/final.contigs.fa.metabat_", "", bin_mapping$bin)
bin_mapping$bin = gsub("\\.fa$", "", bin_mapping$bin)
duplicated_markers$`Bin Id` = gsub("final.contigs.fa.metabat_", "", duplicated_markers$`Bin Id`)
duplicated_markers$`Gene Ids` = gsub("_\\d+$", "", duplicated_markers$`Gene Ids`, perl=TRUE)
colnames(duplicated_markers) <- c("bin", "marker", "contigName")

Create a smaller dataframe of just the binned contigs

binned_contigs = merge(bin_mapping, coverage, by="contigName")

We are interested in Bin 41, lets get just the data for it.

bin_41 = binned_contigs[binned_contigs$bin == "bins_41",]

Since we have three samples we can look at the coverage of the contigs in each sample as a matrix of 2D

pairs(~final.contigs.3730.bam+final.contigs.5133.bam+final.contigs.5579.bam, data=bin_41, main = "Comparison of contig coverage between samples", labels = c("3730", "5133", "5579"))

The majority of the contigs are found in sample 3730 between 10-15x and with very low coverage in sample 5133. Some of the contigs also have some coverage in sample 5579, but most don’t. It’s tempting to remove all of the contigs that don’t fit into the band of coverage, but from this we can’t be certain if the duplicated markers are in these outlier contigs.

Since the coverage of sample 5133 doesn’t really factor into things lets look at a single 2D plot of the 3730 coverage and the 5579 coverage. As a coarse approach lets take a look at where all of the contigs with duplicated markers are in this plot. For starters make a new column in the bin_41 dataframe that tells us if the contig contains any duplicated marker.

bin_41_duplicated_markers = subset(duplicated_markers, bin == "bins_41")
bin_41$containsDuplicateMarker <- bin_41$contigName %in% unique(bin_41_duplicated_markers$contigName)

And now lets plot the data, changing the color of the points based on the value of the ‘containsDuplicateMarker’ column

plot(bin_41$final.contigs.3730.bam, bin_41$final.contigs.5579.bam, pch = 19, col = "lightgrey", cex=0.5, xlab="3730 coverage", ylab="5579 coverage", main="Contigs containing duplicated markers")
points(bin_41[bin_41$containsDuplicateMarker,]$final.contigs.3730.bam, bin_41[bin_41$containsDuplicateMarker,]$final.contigs.5579.bam, col = "red")

Apart from a couple of outliers, the majority of the contigs that contain multiple markers are in the central mass of contigs. There doesn’t appear to be any way to systematically remove a substantial amount of contamination in this genome bin.

Instead of looking at all contigs that contain any duplicated markers, we can also visualise the positions of contigs for a specific duplicated marker. You can get a list of the markers that are duplicated in the bin by using the unique command. Then create a logical vector of bin_41 of the contigs that contain any of the particular markers.

unique(bin_41_duplicated_markers$marker)

##   [1] "PF03439.8"  "PF01172.13" "PF00164.20" "PF09173.6"  "PF03874.11"
##   [6] "PF00203.16" "PF00466.15" "PF00416.17" "PF05670.8"  "PF04127.10"
##  [11] "PF01090.14" "PF01193.19" "PF09249.6"  "PF11987.3"  "PF00298.14"
##  [16] "PF00252.13" "PF01198.14" "PF01201.17" "PF01351.13" "PF04567.12"
##  [21] "PF01922.12" "PF01667.12" "PF00398.15" "TIGR03685"  "PF04983.13"
##  [26] "TIGR03683"  "PF04561.9"  "PF04565.11" "PF04919.7"  "TIGR00425" 
##  [31] "PF01200.13" "PF07541.7"  "PF00679.19" "PF01194.12" "PF13656.1" 
##  [36] "PF01015.13" "PF01282.14" "PF01287.15" "PF00276.15" "PF01780.14"
##  [41] "TIGR00064"  "TIGR00389"  "PF04997.7"  "PF04563.10" "PF03947.13"
##  [46] "PF01912.13" "PF00181.18" "PF04010.8"  "TIGR00549"  "TIGR01018" 
##  [51] "PF01157.13" "TIGR00392"  "PF00347.18" "PF00831.18" "TIGR01080" 
##  [56] "TIGR00408"  "TIGR00270"  "PF08068.7"  "PF00687.16" "PF04560.15"
##  [61] "PF05000.12" "PF00380.14" "PF00752.12" "PF09377.5"  "TIGR02389" 
##  [66] "TIGR00419"  "PF01000.21" "PF04566.8"  "PF00562.23" "PF00366.15"
##  [71] "TIGR00057"  "PF01191.14" "PF00572.13" "PF01981.11" "PF01984.15"
##  [76] "TIGR03677"  "PF00411.14" "PF01864.12" "PF00238.14" "TIGR03724" 
##  [81] "PF00177.16" "PF00827.12" "TIGR02338"  "PF01849.13" "TIGR00468" 
##  [86] "PF01866.12" "PF01982.11" "PF00237.14" "PF03876.12" "PF02978.14"
##  [91] "TIGR00670"  "PF00623.15" "PF01246.15" "PF00573.17" "PF00189.15"
##  [96] "TIGR01213"  "TIGR00344"  "PF01868.11" "PF03764.13" "PF00867.13"

contains_x_marker = bin_41$contigName %in% subset(bin_41_duplicated_markers, marker == "PF01157.13")$contigName

With this vector we can plot the points like we did with the complete set of duplicated markers.

plot(bin_41$final.contigs.3730.bam, bin_41$final.contigs.5579.bam, pch = 19, col = "lightgrey", cex=0.5, main="Position of contigs with PF01157.13", xlab="3730 coverage", ylab="5579 coverage")
points(bin_41[contains_x_marker,]$final.contigs.3730.bam, bin_41[contains_x_marker,]$final.contigs.5579.bam, col = "red")

So now we can see that ther are three contigs containing this marker, one appears to be an outlier, but the other two contigs have quite similar coverage profiles

The pace of genome binning from metagenomes

c.skennerton@gmail.com (Connor Skennerton) — Wed, 17 Jun 2015 00:00:00 +0000

With the pace of science what seemed top stuff three years ago is now an order of magnitude less than what just got published.

When I was in my PhD I was working on EBPR communities using metagenomics to characterize the the microbial and phage populations. From all of my microbial data I could get about 70 draft bacterial genomes. At the time (2012) that was pretty huge considering the first big paper on genome binning from metagenomes got 49 draft bacteria (Wrighton et al, 2012). The next year a similar study from EBPR recovered 31 draft genome bins (Albertsen et al, 2013). One of the big results from all this computational effort is that a number of genomes from candidate phyla. In 2012 I was super excited that I had multiple genomes from Saccharibacteria and Microgenomates which at the time had only a few genome representatives. At the time I thought I had the richest dataset in the world (which I most definitely did not) now three years later and ~800 genome bins just got published mostly from those candidate phyla I just mentioned (Brown et al, 2015). It seems like the magnitude of the cutting edge just went up a notch…

Genome Bin Completeness and Contamination

c.skennerton@gmail.com (Connor Skennerton) — Thu, 29 Jan 2015 00:00:00 +0000

The question that most people ask when looking at a metagenomic draft genome bin is: should this gene really be there? The answer is that sometimes it’s not easy to know

I spend a lot of my time looking at genome assemblies. They are almost always from metagenomic data and usually are from some novel phylogenetic lineage with few (if any) “close” relatives. Unfortunately the quality of genome assemblies is such that all of these genomes are in many pieces and due to the nature of binning, there is always the spectre of some genes not belonging to the genome that your analyzing. I’m currently a middle author on a paper that attempts to quantify genome completeness and contamination using single copy marker genes. The idea is that there are genes that are always found in a single copy per genome, therefore if you find them in multiple copies then you’ve got some contamination from other sources. This is great in principle but the number of single-copy genes is going to be very small for a large set of genomes, for example there are about 100 of these genes that are common for all bacteria. Assuming that some bacterial genome has approximately 3000 genes, then looking at only 100 of them is about 3% of the gene content. That’s quite a small percentage to be making big statements on how good or bad a genome is.

So here is a question that one of my colleagues posed in a group meeting: if a genome has 10% contamination does that mean that 1 in 10 genes shouldn’t be there?

My answer is: of course not, it means that 1 in 10 of the marker genes are in multiple copies. But it does get me thinking, how can we tell that 10% of all genes in the genome aren’t from contamination?

One of the observations with single-copy marker genes is that many of them are co-located such that getting one erroneous contig can disproportionatly increase contamination. Similarly all of the genes of a contig are either correctly or incorrectly binned, rather than every 10th gene in some contig being from a different genome. If we were to assume a perfect situation where every contaminating contig contained at least one marker gene then the real value of contamination would be the total number of genes on those contaminating contigs divided by the total number of genes in the genome bin. The “real” contamination percent then becomes more about the size of the contigs rather than the number of marker genes. This is easy in theory but hard to determine in practice since you need to know which copy of the marker gene is the contaminant and which is legit.

It’s easy to remove contigs with extra marker genes to lower the contamination numbers but what about short contigs that contain no markers? I don’t think there is a perfect way to tell if they belong to a genome or not but one approach has been to look for paired reads that link these contigs to larger ones in the assembly even if they are not contiguous. Here again there are questions raised, if there are reads that link the contig to the bigger assembly why is it separate, why hasn’t the assembler joined the contigs together? The magic behind genome assemblers creates many interesting outcomes that I can never explain and as such there is no general rule for believing paired-end links or not. Honestly, I usually go with my gut (very unscientific) and generally only bother to go further in depth if there are genes that are important to the story I’m trying to write.

The golden rule of interpreting genome bin assemblies is that the bigger the contig the more confident that you become in assigning them to a draft genome. (Of course the answer is just getting a better assembly :P).

With these observations and caveats in mind, what value is there in placing a percent completeness or contamination based on marker genes if it doesn’t really relate to the total gene content?

I think there is a lot of value but not when the numbers are taken literally. Marker genes are an approximation and a way of sorting genomes that are worth a closer look. From experience I can tell you that a genome that is “90% complete, 5% contamination” looks a lot better than one that is “50% complete, 30% contamination”. I take these numbers to be a relative score along with other factors like the total number of contigs (less is better) and the total number of bases (does it look like it’s about the size of a whole genome) to point me in the right direction of what I should work on first.

Uploading your Theis Literature Review to Wikipedia

c.skennerton@gmail.com (Connor Skennerton) — Wed, 12 Nov 2014 00:00:00 +0000

Every PhD student can contribute to open science by uploading their thesis literature review onto Wikipedia!

My thesis was entitled “Phage-host evolution in a model ecosystem”, where I tracked the evolution of phage genome evolution and the evolution of bacterial defense mechanisms using metagenomics. When I was writing my thesis I spent a lot of time writing up the section on CRISPRs, which are a type of bacterial adaptive immune system. The CRISPR field has exploded in recent times due to its applications for genome editing which has also meant that there have been numerous great reviews on the ever expanding primary literature. Despite this though I found the CRISPR page on wikipedia to be a bit bare (of course now that I’ve updated it things look better, but check out the edit history for a look at what I’ve added). Since there was already a saturation of reviews on the subject I decided to go through and add in what I had written for my thesis to wikipedia. By far the most important and difficult part of this is was formatting the references right. So I made a simple workflow for getting what I wanted out of my thesis and into wikipedia.

First thing, I wrote my thesis in word and used endnote to format my references using the style from nucleic acids research (which is a numbered style). If you want to use my method you need to have the exact formatting style
Export the literature review from word as plain text and then removed all the parts that aren’t needed
Put back in some formatting for the headers using wikipedia’s markup
Split out the bibliography into a separate file
Then use some very dodgy perl code I wrote to format all of the in text citations into the format that wikipedia requires

This perl code takes the text and looks for numbers, or lists of numbers inside parentheses. For example: (1,2,5), (1-7,14), (1). Then looks up what those references are in the bibliography, extracting the title of the journal article. Then it looks up what the pubmed ID is for that paper using entrezdirect with the title as the search term. After getting the pubmed IDs for all the cited references it saves them to a file, so that it can be used later on (without looking up the pubmed IDs again) and any errors can be fixed manually. Finally all of those in text citations are replaced with the proper wikipedia markup and your ready to upload.

There are so many scientific wikipedia pages that are way too brief for their subject matter and I’m not just talking about the thousands of stubs for bacterial and archaeal species either. My current work is with methane seep sediment communities that perform anaerobic oxidation of methane (AOM), the wikipedia page contains five paragraphs. Looking at the broader topic of methanogenesis there is still only three short paragraphs on the biochemistry. We know so much more than this, all that knowledge fills the pages of hundreds of review articles yet when someone, anyone types anything into google the first match that comes up is wikipedia. We should make it better cause it’s ultimately the most visible publication on the internet.

My First Useful Script

c.skennerton@gmail.com (Connor Skennerton) — Mon, 23 Jun 2014 00:00:00 +0000

Can you remember the first useful thing that you ever coded? I sure can, and I’m thankful for it every day.

I’ve recently finished writing a little program called fxtract (which I’ve blogged about before) that acts like grep but returns whole fasta or fastq records from a file. It’s taken me a very long time to write this thing, primarily cause I’m writing it in C++ but now that it’s pretty much done I’m feeling nostalgic about why I’m writing it in the first place.

Back in 2010 I was just starting my PhD with Gene Tyson and learning some bioinformatics on a “toy” dataset (which became my first paper) that was a huge 10 Mbp of viral sequence data! At the time I was learning Perl using this great online tutorial but I found that I needed some more “real-world” examples to really get to grips with the language. I can’t fully remember what I was doing but I had a tabular blast output file and I wanted to get the sequence of some contigs with blast hits. I jumped at the opportunity to solve this with my nasent coding skills and so I started writing my first useful bit of code: contig_extractor.pl.

In the beginning it did only what I first wrote it to do: take a blast file and a fasta file and return some contigs. Over time though it morphed into something more general - a way of getting some subset of a fasta file using a list of identifiers. Over the years I added functionality for different file formats and allowed searching using regular expressions. This one piece of code eventually disseminated throughout the whole lab group as new people came in and needed to solve the same problems. I think that the introduction to github for many of the PhD students who came after me was downloading my random collection of scripts just so they could get their hands on contig_extractor.pl.

Of course every bioinformatician has probably written the same piece of code to solve the same problem. If I had been smart enough to look I’m sure that I would have found one, but using a script doesn’t quite give the same level of satisfaction as writing it yourself (and finding that it’s useful by all your colleages).

There came a time though when contig_extractor.pl reached it’s limit. I was trying to extract a few thousand records from a fastq file with 100 million and Perl just wasn’t cutting it anymore. And so fxtract was born to do all the good things from contig_extractor.pl just a heap faster.The sun is setting on my first useful piece of code, at the moment only my finger memory and force of habit keeps it being used. But when I think about it, it’s been an amazing four year run for something that I probably thought was going to be a one-off script, hopefully fxtract can get as much love from me as well.

Finding 16s/18s reads in metagenomes

c.skennerton@gmail.com (Connor Skennerton) — Sat, 19 Apr 2014 00:00:00 +0000

Got a Metagenome? want to know what the community looks like?

rRNA operons are typically poorly assembled in metagenomic datasets due to highly conserved sequences. More targeted assembly approaches may be necessary to obtain accurate reconstructions from short read datasets. There are a few ways in which we can extract reads originating from either 16S or 18S reads and there are a number of programs (SSU-ALIGN, rRNASelector, riboPicker, SortMeRNA, blast, bowtie or bwa) to name a few.

There are a other ways of looking at the communty from raw metagenomic reads. These are mostly kmer-based approaches like kraken or extended marker gene approaches like MetaPhlAn or phylosift. I’m not against any of them, just never used them, so I’m going to keep the following post specific to 16s/18s community composition

Extracting 16S/18S reads

I’ve used SortMeRNA and bowtie/bwa in my workflows with good success. The difference between these two methods is that SortMeRNA that uses a kmer searching method using an index created from a database of previously sequenced genes whereas bowtie/bwa use a local alignment method to compare the query sequence to a previously made database. Below are instructions on how to use both methods.

Installing and using SortMeRNA

From the source page for SortMeRNA on github copy the ssh clone url and then open up a terminal window and type

$ git clone git@github.com:biocore/sortmerna.git
$ cd sortmerna
$ ./configure --prefix=$PWD
$ make install

Mac OSX install note: I found that there was an error with my install with configure complaining that it was missing install-sh. I solved this by copying autogen.sh into the same directory as configure, execute it and then try running configure again.

The commands above should install SortMeRNA in the directory which you downloaded it to.

Building indexes for SortMeRNA

SortMeRNA comes packaged with 8 different rRNA databases, all derived from SILVA. Build either the 16S or 18S database, depending on what you want. Below is the command used to generate the 18S index (assuming that you’ve installed in the download directory)

$ bin/indexdb_rna --ref rRNA_databases/silva-euk-18s-database-id95.fasta,index/silva-euk-18s-database-id95 --sensitive

If you ls the index directory you should see four files generated.

Preprocessing reads

Now we can use this index to extract the reads that may come from 18S using the sortmerna command. I’m going to assume that there are files containing raw reads from an Illumina run called file_R1.fastq.gz and file_R2.fastq.gz. Unfortunately one of the limitations of SortMeRNA is that it requires that you only give it a single file and that the file is uncompressed. So to start with, unzip the files using gzip and then combine them into a single file using merge-paired-reads.sh found in the scripts directory of the SortMeRNA source code

$ gunzip file_R1.fastq.gz file_R2.fastq.gz
$ bash scripts/merge-paired-reads.sh file_R1.fastq file_R2.fastq combined.fastq

To save space, it’s probably best to re-zip the files to save on harddrive space

$ gzip file_R1.fastq file_R2.fastq

Extracting the reads

Now we can run sortmerna, saving matched reads (and their mates) to a file with the prefix ‘matched-18S’ in fastq format using 4 processors.

$ bin/sortmerna --reads combined.fastq \
    --ref rRNA_databases/silva-euk-18s-database-id95.fasta,index/silva-euk-18s-database-id95 \
    --paired-in \
    --fastx \
    --aligned matched-18S \
    -a 4

Installing and using BWA

My personal preference is to use bwa over bowtie but the merits of either are debatable (there are also 50 other programs out there that try to solve the same problem so the choice is yours). Download bwa from github by copying the ssh clone url and typing the following into the terminal. (Change directories out of the SortMeRNA source directory before you do this)

$ git clone git@github.com:lh3/bwa.git
$ cd bwa
$ make

bwa also requires that an index of the database be made. For simplicity, lets use the same database that was included in SortMeRNA. To make the bwa index copy the files that you want from the SortMeRNA source directory into a new location.

$ cp ../sortmerna/rRNA_databases/silva-euk-18s-database-id95.fasta .
$ bwa index silva-euk-18s-database-id95.fasta

This should make a whole bunch of files that have extra file extensions appended to the fasta file. With the index created we can now align the reads

$ bwa mem silva-euk-18s-database-id95.fasta file_R1.fastq.gz file_R2.fastq.gz > aligned_18S.sam

(Notice that there is no preprocessing steps necessary for the reads)

Postprocessing the sam file

The output of bwa is a standardized format called sam which many programs will now output. This format essentially describes the alignment of each of the query sequences to the reference sequences. This is not exactly what we want, which is the reads that were successful hits to the reference in fasta/q format. To go from a sam file to a fasta/q file is a little complicated (I wish it wasn’t). To start with, download samtools and fxtract from github, and download them into new source directories (bonus points for getting them installed without a walkthrough). First convert the sam file into its equivalent binary format and filter out unaligned sequences:

$ samtools view -SubF 4 aligned_18S.sam | samtools sort - aligned_18S && samtools index aligned_18S.bam

Now get the names of all the reads that aligned:

$ samtools view aligned_18S.bam | cut -f 1 >aligned_reads.txt

And finally extract those reads from the original fastq files:

$ fxtract -Hf aligned_reads.txt file_R1.fastq.gz file_R2.fastq.gz >aligned_reads.fastq

Poking around inside grep

c.skennerton@gmail.com (Connor Skennerton) — Fri, 27 Dec 2013 00:00:00 +0000

Playing around with the grep source code to make it output fasta/fastq records. Check out the code here.

I’m quite interested in string searching algorithms as I’ve written a program, crass, which uses a few of them to search for CRISPR elements. Crass is pretty fast, but I want it to be faster, specifically there is one point in crass where it searches for exact matches to many thousands of patterns. In a previous post I tried out a number of different ‘multi-pattern matchers’ from seqan and was fairly unimpressed with their speed. In this testing though I did not try out the most widely used implementations of multi-pattern matching: GNU grep (using -Ff options). I don’t have any graphs to show, but it was faster, like a lot faster.

It wasn’t a fair fight though, as grep works very differently to my test program fxtract. There is a great mailing list post by one of the original grep authors on ways to speed things up. It basically comes down to I/O (not that the search algorithms aren’t highly optimised themselves), fast input from a file and not copying the data in memory is the key. Grep doesn’t try to parse the file instead it just loads it into a big buffer in memory and searches the whole thing, if it finds a match then it figures out the boundaries of the line the match is on and prints it out. This is in contrast to programs like fxtract or crass that parse the file first to get the individual portions of each record before the search is performed.

This got me thinking that I could drastically speed crass up if I switched over to the grep way of processing files. I was a little worried though that determining the boundaries of a fasta/fastq record from an anonymous buffer might be a bit tricky so before I modified crass I chose to modify grep so that it would output fasta or fastq records.

The printing functions in grep live in main.c and start with the function grep, which in-turn calls grepbuf, which in-turn calls prtext etc. The code is surprisingly simple, grep reads from the file and fills a buffer; grepbuf executes one of the search functions on that buffer; if a match is found, a pointer to the first character in the line the match was found is returned; and then the printing functions take over. The printing functions get a pointer to the start of the line and the end of the line of the match and pass that through to fwrite. Everything is handled using pointer arithmetic for determining the start and end of where to print.

This is great since it’s easy to change the pointer to the beginning a end of a record, rather than a line. So that just left the logic for me to write in to find the limits of a record. Below is a code snippet from grep where I’ve added in the logic. Fasta is easy to implement as the > symbol is generally unique. Fastq on the other hand takes a bit more work, since the @ symbol can also be found in the quality string.

char const *b = p + match_offset;  /*pointer to beginning of matching line*/
char const *endp = b + match_size; /*pointer to end of matching line*/
/* Avoid matching the empty line at the end of the buffer. */
if (b == lim)
  break;
if(fasta_input)
  {
    /*find the beginning of the record*/
    while(b > p && b[0] != '>') --b;
    /*find the end of the record*/
    while(endp < lim && endp[0] != '>') ++endp;
  }
if(fastq_input)
  {
    /*find the beginning of the record*/
    while(b >= beg)
      { 
        if(b[0] == '@')
          {
            if(b - 2 <= beg)
              /*can't go back any further therefore must be start of record*/
              break;

            if (b[-1] == '\n' && b[-2] != '+')
              /*@ symbol at beginning of line but not the first in the quality */
              break;
          }
      --b;
    }
    endp = b;
    int newline_count;
    for(newline_count = 0; newline_count <4; ++newline_count)
      {
        /*find the end of the record*/
        while(endp < lim && endp[0] != '\n') 
          ++endp;
    
        ++endp;
      }
  }

The fastq format parsing has fairly obvious corner cases since with this code there can be no text on the ‘comment’ line and the whole record must be of four lines. This version of fastq is the recommended formatting from the official fastq publication, which seems to have been adopted by Illumina and others, so hopefully this simple parsing will work most of the time.

It’s been a heap of fun looking at the way this very mature piece of software works and I’ve gotten a usable tool out of it. Now it’s onto the main event of ripping out parts of the source code that I want for crass

Genome Scaffolders Suck

c.skennerton@gmail.com (Connor Skennerton) — Wed, 11 Dec 2013 00:00:00 +0000

Experiences using a variety of contig scaffolding tools. It was not a good experience.

Recently in our lab we’ve been getting some Illumina mate-pair data to improve some metagenomic assemblies. The sequencing has been going well and we’ve been generating a good number of mate-pairs without too much duplication, but we’ve had quite a bit of trouble with the bioinformatic part of actually using this data to improve the assemblies. There are a number of software tools available to link contigs together after an assembly has been done, however many assume that you are scaffolding a genome not a metagenome.
One of my colleagues suggested that, as a group, we perform a comparison of these programs to determine which is the best one to suit our needs.

We decided to test out sspace, grass, scarpa, opera, mip and bambus2. From these six programs, only bambus2 was designed for metagenomes. To test how well genome scaffolders could transition into metagenome scaffolders we took one of our metagenomes and randomly broke up 1000 contigs formed only using paired-end data into two pieces and attempted to use some shiny new mate-pair data to put the pieces back together again. Little did we know the world of pain that we were about to step into…

The following post will not talk about which algorithm was better or even really what results each of these programs gave. No, I’m going to talk about why using all these programs sucked.

The war of attrition

grass

We didn’t even get grass installed; it required a particular library distributed by IBM, which while free for academic users, required you to give over some personal details, which we decided was something that we didn’t want to do.

One down, five to go…

mip

Oh where do I start with mip! Mip assumes that you’re using SOLiD data, which means that it expects that the orientation of the reads are forward-forward. Since Illumina’s mate-pairs are reverse-forward I needed to reverse complement the first read, making a duplicate file. But actually I needed to modify the second read file as well because mip needs to have the read identifiers ending in “_R3” or “_F3” (WTF!), just like they would be for a SOLiD run. After this, I could then run any short-read mapper to produce a SAM file for further processing. But wait, you have to map each read separately (i.e. not as pairs) and then pass the SAM file through two preprocessing shell scripts. The first of these shell scripts, the opaquely named merge.sh simply takes the corresponding SAM alignment record from the first and second reads and concatenates the lines! Why this needed to be a separate script and not just part of the software proper is beyond me. Both of these preprocessing scripts created new files that were essentially duplicates of the original SAM files made by the short-read mapper, which I truly find annoying, especially since our server is currently short on storage space.

When I ran mip proper, it threw an error. That error may have been easy for me to fix, it might have been something silly that I had put into mip’s configuration file, I don’t know. At this point I was pissed about the whole process and never tried to run the program again.

Two down, four to go…

opera

It segfaulted on the test data that came with the software.

Three down, three to go…

bambus2

I had hope for bambus, and this was the scaffolder which I put the most amount of effort to get it to work. The problem with bambus is that it is part of the AMOS software package and the problem with AMOS is that it is fairly old and uses it’s own file format. I have no doubt that the authors of AMOS were hoping that their file format would catch on and that many tools would plug into it, but alas the AMOS file format did not catch on, instead we have SAM files. So I set about looking for a SAM to AMOS file converter, and I found one in the AMOS package. It was a nice little perl script that unfortunately didn’t work (once again threw an error).

I decided to write my own converter in C++ using the bamtools and AMOS C++ APIs. The fruits of my labour was the program sam2amos, which can take a BAM file and convert it into either an AMOS message stream or an AMOS bank. I was honestly really impressed with the simplicity of both of these APIs, which made writing this program relatively easy.

After making my AMOS bank file with sam2amos I ran the goBambus2.py wrapping script, which unfortunately resulted in bambus segfaulting. Bugger! I didn’t have it in me to debug the situation and figure out whether it was caused by a problem in sam2amos (which it probably is) or something else.

Four down, two to go…

Finally some results

scarpa

I never personally ran scarpa (one of my colleagues did) but it shared some of the annoyances that mip had. Scarpa assumes that the reads are in forward-reverse orientation so it required that both of the reads be reverse complemented before use. Like mip, the reads needed to be mapped independently. But unlike mip, scarpa gave some results!! Of course the output was a simple fasta file of scaffolds without any indication of what contigs were actually put together. A little more information would have been nice…

sspace

Sspace is the best of the bunch when it comes to usability. You can specify what orientation the reads were in and there was only one manual preprocessing step required; sspace does not accept compressed files so the reads had to be passed through ungzip before use. Sspace does however have some annoying automatic preprocessing steps. First, the reads are copied into the output directory (why? I don’t know). Second, sspace only runs bowtie, you cannot specify any other mapper, this isn’t so much of a problem but it annoyingly repeats the mapping even if the results from a previous run exist in the output directory. The output though is better than scarpa’s as it gives you a fasta file of scaffolds, an “evidence file” actually telling you how it put contigs together and optionally a graphviz file so that you can look at the contig linkages.

What I want from a scaffolder

The ability to take either a SAM or BAM file of read mappings; or
If a particular read mapper is really required, give me the ability to set the read orientation and used commpressed input files
One that uses pipes and does not copy reads into an output directory. The less file I/O the better
No configureation files, I want to pipeline this thing, give me some command-line options
Tell me how the scaffolding was done with some sort of easy tabular file format. Better yet make that output format AGP, which is required by NCBI when uploading scaffolds.

Epilogue

Oh yeah, sspace gave better results than scarpa

Testing out Seqan's Multipattern Search Implementations

c.skennerton@gmail.com (Connor Skennerton) — Mon, 28 Oct 2013 00:00:00 +0000

I recently discovered Seqan, a header-only C++ library for bioinformatics. I’ve been playing around with the toolkit to make some small programs just to see whether I want to use it in a larger project. So far I’ve written prepmate, an adaptor trimming program for Illumina’s Nextera mate-pair libraries; and fxtract, a grep-like program for extracting fasta/fastq records from large files. One of the algorithms that I use in fxtract and in another program I’ve written, crass, is to search for multiple patterns simultaneously (in this case a number of different DNA motifs). Seqan implements a number of algorithms for multipattern matching (checkout their tutorial page), however they don’t give many clues as to why you would choose one algorithm over another. So I decided to take a few of these implementations out for a spin using fxtract.

Test variables:

different numbers of patterns: 100, 1000, 10000
different length patterns: uniformly distributed size (0 - 100bp), 30bp, 60bp
Algorithms: WuManber, MultiBfam, AhoChorasick, SetHorspool
Test dataset: SRR438796, which is part of an EBPR metagenome that was sequenced with Illumina GAIIx

I should point out the the following speed tests are not necessarily testing the algorithms, but the seqan implementation of these algorithms in the context of what fxtract is trying to accomplish. Which means that other programs which implement these algorithms may perform quite differently and the conclusions that I reach below may be different in other situations. So with disclamers out of the way, how does each implementation perform.

While running through a test of the wumanber implementation I discovered something very odd when using the uniformly distributed patterns. With 100 patterns the program ran through fine, outputting the correct results, however when using the 1000 or 10000 pattern file I got no output. I discovered though that these larger files contained some patterns that had a length of 0 (i.e. they were empty lines), whereas the 100 pattern file’s smallest pattern was 6bp. I removed the empty lines from my pattern files and then and voila there was output. What this means is that the seqan WuManber implementation fails silently when given an empty string as one of the patterns! I’ve submitted a bug report with the seqan authors.

Seqan speed test summary

The above graph shows the total wall clock time in seconds for each algorithm to process the input file with the different numbers of search patterns (columns) or for the different types of patters (rows).
The main thing that striked me was the poor performance of the
seqan implementation of the Wu-Manber algorithm - it does not scale well.

I was lead to believe that Wu-Manber was the pinicle of multipattern searching so I was a little perplexed by this result. I know of another stand alone implementation of Wu-Manber provided by Ray Burkholder. As a comparison I tested out this implementation as well and perhaps unsurprisingly this implementation performed much better (it’s referred to in the figures as “non-seqan wu-manber”). Bottom line: So don’t use the Wu-Manber implementation in Seqan! The winner of the speed benchmark was the multiBfam algorithim which performed nearly identically for any number of input patterns or for the different length patterns. In comparison the Aho-Corasick and SetHorspool algorithms became slower with more patterns.

Speed tells you only half of the story and usually comes at the price of memory consumption. Both of the Wu-Manber implementations excelled at memory consumption, or lack there of. The worst were the Aho-Corasick and SetHorspool algorithms, which used 20Gb+ of memory under the worst case. The multiBfam algorithm was somewhere in the middle (~7Gb under the worst case scenario).

Seqan speed test summary

Considering only the seqan implementations I would probably go for the multiBfam algorithm as my first choice when searching for less than 1000 patterns after which Wu-Manber would have to be preferred.
Base on what I’ve seen the Aho-Corasick and SetHorspool implementations in seqan are the worst options as they use far too much memory, however that’s not to say that other implementations of these algorithms would perform differently.

Scripts

make_patterns.sh

Running this script requires that both seqtk and bioawk are installed and in your PATH.

for count in 100 1000 10000
do
    # make uniformly distributed patterns
    seqtk sample SRR438796.fastq.gz
$count |\
        bioawk -c fastx '{
            start=rand() *99;
            end=rand() * 99;
            if(start < end){                 
                if(end - start > 15) {
                    print substr($seq, start, end)             
                }
            }else{
                if(start - end > 15) {
                    print substr($seq, end, start)
                }
             }         
}' >random_${count}_patterns.txt

    # make 30bp patterns 
    seqtk sample SRR438796.fastq.gz
$count |\
        bioawk -c fastx '{
            start=rand() * 30;
            end=start + 30;
            print substr($seq, start, end)
        }' >random_${count}_30bp_patterns.txt

    # make 60bp patterns
    seqtk sample SRR438796.fastq.gz
$count |\
        bioawk -c fastx '{
            start=rand() * 30;
            end=start + 60;
            print substr($seq, start, end)
        }' >random_${count}_60bp_patterns.txt
done

####time_implementations.sh

#! /bin/bash
for impl in fxtract-*
do
    for p in speed_test/random_100*
    do
        patterns=$(basename $p)
        echo "real,user,sys,avemem,maxmem" >speed_test/${impl}_${patterns%.txt}_times.txt
        for i in {1..10}
        do
            /usr/bin/time -f "%E,%U,%S,%K,%M" -a -o speed_test/${impl}_${patterns%.txt}_times.txt ./$impl -f $p SRR438796.fastq.gz |\ 
                wc -l >>speed_test/${impl}_${patterns%.txt}_words.txt
        done
    done
done

summarize_times.py

#!/usr/bin/env python
from __future__ import print_function
import sys, os, glob, re

fixed_re = re.compile('fxtract-(\w+)_random_(10{2,5})_(\d\d)bp_patterns_times\.txt')
norm_re = re.compile('fxtract-(\w+)_random_(10{2,5})_patterns_minus_small_times\.txt')

impl_codes = {'ac': 'aho-corasick', 'sh': 'set horspool', 'mb': 'multi-bfam', 'wm': 'wu-manber'}

times_files = glob.glob("./fxtract*times.txt")
data = {}
for fn in times_files:

    match = fixed_re.search(fn)
    if match:
        impl = impl_codes[match.group(1)]
        pattern_type = match.group(3)
        pattern_count = match.group(2)
    else:
        match = norm_re.search(fn)
        if match:
            impl = impl_codes[match.group(1)]
            pattern_type = 'norm'
            pattern_count = match.group(2)
        else:
            raise ValueError("Filename %s does not fit either form" % fn)
    with open(fn) as fp:
        first_line = True
        seconds = 0.0
        for line in fp:
            if first_line:
                first_line = False
                continue
            fields = line.rstrip().split(',')
            seconds += float(fields[1]) + float(fields[2])

        seconds /= 10.0
        print(impl,pattern_type, pattern_count, seconds, sep="\t")

plot_summary.R

library(ggplot2)
data = read.table("summary.txt", header=T, sep="\t")
p <- ggplot(data, aes(Algorithm, Wall.time)) + geom_bar(stat="bin")
png("seqan_multipattern_search_speed_benchmark.png", width=600,
height=600)
p + facet_grid(Pattern.type ~ Number.of.patterns) + theme(axis.text.x =
element_text(angle = 45, hjust = 1))
dev.off()