I’ve previously had around 3 years experience working for a fairly small startup in the fintech space. While Capchase is small, the numbers are no joke and the amount of work was also no joke.
I mainly did work with the backend, data processing, infrastructure, CI and rarely I also did some frontend work.
What I did:
Migrating services from Python to Elixir
The first project I tackled when I joined the company was recreating data fetching systems that already existed in Python into the new Elixir system that was being built at the time
Integrating new Data Sources
During my time at Capchase I ended up integrating with a lot of services, some of them were Teller,, Tink and others.
Integrating a service implied creating all the logic to fetch the data from the APIs daily without wasting resources (don’t fetch more data than you need to), sending the data to BIgQuery, creating the pipelines to transform the data for the new service, creating the UI for connecting that integration in our system.
I also did a lot of maintenance work on all the integrations – any time they would suffer a breaking change, deprecate previously available features, etc.
Creating a Data Integrity System
Even though we had a lot of alerting and metrics in place we were still a little paranoid that we could be losing some transactions or random data at some point in the pipeline due to, for example, a misplaced filter… So we created a system that allowed us to compare the data directly from the data sources with our processed data from BigQuery, telling us if we were any transactions, if the balance was updated, if the number of accounts is correct, etc… This system ran daily and outputted the data both to looker but also had a frontend part that allowed us to investigate each issue in detail.
Creating Looker Visualizations
At some point we adopted Looker into our list of services we use. I was tasked with creating Visualizations that allowed us to understand better the relation between the customers and the types of data sources they were likely to connect to our system. The objective was to try pushing more customers to connect more sources. I created a lot more visualizations, mostly for the company’s internal data and for visualizations related to the aforementioned integrations
CI Improvements
This one is particularly important to me, we had a monorepo with a bunch of services and we also had separate repos with smaller services. Any new commit to main must come from a PR merging. This means that for every PR you need to run a lot of checks, formatting, linting, coverage, tests passing, etc… And improving the speed of the workflows was something very satisfying as it is something you, as a developer, deal with day in day out.
To improve this we tried to add as much caching as possible, we managed to split the tests so they could run in different runners, we created multiple workflows one for each specific task so that all could start in parallel. Nothing beats the feeling of reducing the 20 minutes pipeline to 7 minutes.
New Data Processing Platform
At some point we decided we wanted to start using our own internal system to do the data transformations, this was so that we could save on BigQuery costs as the costs were getting higher and higher the more data we added.
For this we created a Python system using PostgreSQL to store the raw data and then using duckdbt to create pipelines transforming that data in memory and saving it to an S3 bucket.
This came with lots of challenges that needed to be overcome, connectivity problems, getting the initial data from the production BigQuery tables to our PostgreSQL database, Out Of Memory problems (BigQuery having infinite RAM truly helps it’s performance), async issues, just a whole load of problems you would get when trying to create such a system. But in the end we learned a lot and it was very rewarding to see the system working in the end.
Daily Maintenance
From time to time we would see alerts that needed attending, a new feature that is raising errors, a new integration that broke a pipeline, a new migration in the data processing platform is preventing the system from booting, etc… Those were all issues that I was taking care of, trying to keep problem out of the production environment, some times the solution was rolling back and reverting migrations, other times it was creating a quick hot fix and other times it was disabling a feature flag.
A lot more
The truth is that when working in a startup you end up getting involved in a little bit of everything so at some point I worked on the frontend, backend, infrastructure, CI, ETL, anything that was tech related at the company.
This was further fomented by the company having Rotations, where the some tech people would do the role of supporting the rest of the company. I did a lot of Rotation weeks where it was asked of me to fix issues in systems I had never touched before, and, by persistence and asking around, I was able to solve them.
What I learned:
Working under pressure
In the weekend SVB went out of business the company started intaking a lot more new leads than usual, that combined with some inefficiencies of the data pipeline were making it so the data was taking a lot more time to process than usual and that was affecting our customer experience. At that point, to attempt and improve the issue I started to try to locate what was taking a long time in the pipelines, I saw that new stuff had been added recently that was taking longer than expected and it belonged to a project that was not yet in use so we did some surgery and nipped part of the pipeline off, released a new hot patch version and improved our intake speed significantly!
There’s always more work – always something to improve
Technical Expertise – if you keep learning about the systems you are working on you eventually attain a lot of knowledge about dealing with those systems
We were using Elixir at Capchase to create daily jobs that would fetch data from certain sources and store that data in BigQuery. Elixir is super efficient and we only had like 2-5% usage on our CPUs and at some point that became 90% to do the same tasks… I went digging and ended up using the profiler to read where the problem issue might be. The knowledge I learned from that experience was valuable enough that I ended up writing a tech article for the company’s Medium page about the whole thing.
Write everything down
those commands your colleague just sent you to run are going to be needed and you will be spending a lot of time searching slack for commands unless you save it all in one place…
If everything is written down there is no doubt for anyone reading whatever thread it may be what the state for it would be.
Proposals/Post-Mortems/Document Aided Decisioning
If you start using these documents the way they are intended and really follow through with all the specifications, comments and action items, I believe that these documents are a great tool to really improve the quality of a system. Democratizing knowledge of all important features to everyone.
Technologies:
Elixir
Python
NATS
docker compose
Github Workflows
Kubernetes
dbt
duckdbt
BigQuery
Looker
poetry
pyenv
dagster
Cypress
pm2
ReactJS