Story of Consistent Hashing with a secret sauce.

Etsy, uses memcached (to cache db queries, expensive calculations etc) and Varnish (to cache internal HTTP requests) to improve performance and reduce load.

They have a pool of caching servers, each running memcached or Varnish.

How to distribute Cached data across Servers?

1. Modulo Hashing

It is implemented as follows

1. Hash the key to a number

2. Modulo it by the size of cache pool to store key value pair to a specific server

Example

• We need to store some_key and some_value.

• We have 3 host servers numbered as 0, 1, 2

• We map some_key to a number (via a hash function) let's say number is 4

• Taking modulo 4 % 3 == 1, hence we will store some_key to "1" server

Pros

• Each cached value is stored on a single host

Cons

• Changing the size of the cache pool will cause most cache keys to hash to a new server leading to tons of cache misses.

  • In above example, if we add a fourth server.

  • Taking modulo 4 % 4 == 0, some_key will now be cached at "0" server.

  • Worsening the cache hit rate

2. Consistent hashing

It is implemented as follows

1. Fix the Hash space and divide it into large contiguous buckets

2. One (or more) buckets are assigned to a cache host

3. Keys that hash to a Range are looked up on the matching host

Example

• Let's say, Hash Space = 1,000

• We divide it into 3 buckets of [0, 333), [333, 666), [666, 999]

• We have two hosts,

  • host 0 is assigned two buckets viz [0, 333), [333, 666)

  • host 1 is assigned one bucket viz [666, 999]

• We need to store some_key and some_value.

• we map some_key to a number (via a hash function) let's say number is 44.

• As "44" falls in first bucket range, it is stored in host 0

Pro

• Changing the pool size means moving few buckets from existing server to a new server. Hence it moves only a small number of Keys

  • In above example, if we add another server host 2 and move [333, 666) bucket to it

  • Only 333 keys will be moved to a new server i.e. 33% of key movement.

  • We can play around with bucket size to further decrease key redistribution.

They leverage Ketama to implement Consistent Hashing

How to serve Hot keys?

Certain cache items are read and written more than others.

Ideally, a good hash function shall distribute the "hot keys" among many servers

"Celebrity" Key

At Etsy, they've seen keys that are hit quite often, and stores a large enough value, that it saturate the network interface of their cache host.

The large number of clients are collectively generating a higher read rate than the cache server can provide.

To fix this they explored following solutions

Solution 1: Horizontal Scaling

Adding more cache hosts doesn't help because it only changes the distribution of keys to hosts i.e. it would only move the problem key and saturate a different host

Solution 2: Faster network cards

It will have a substantial hardware cost.

Solution 3: Cache Smearing

They used consistent hashing and added a small amount of entropy to certain hot cache keys for all reads and writes.

This effectively turns one key into several, each storing the same data and letting the hash function distribute the read and write volume for the new keys among several hosts.

Example

• Let's say celebrity_key is a Hot Key

• Cache smearing appends a random number in a small range (let's say, [0, 5)) to the key before each read or write. So for e.g. celebrity_key will be transformed as celebrity_key_1, celebrity_key_3, celebrity_key_0

• As celebrity_key_1, celebrity_key_3, celebrity_key_0 keys are different, they will be hashed to different hosts, sharing the load among the pool.

Tradeoffs in choosing the range of random number

• Too-large range duplicates the cached data more than necessary to reduce load, and can lead to data inconsistencies

• A too-small range doesn't spread the load effectively among the cache pool (and could even result in all the smeared keys still being hashed to the same pool member)

Conclusion

In summary, combination of consistent hashing and cache smearing has been a great combination of efficiency and practicality

Allowing multiple hosts to serve requests for a Hot key, prevent any one host from being overloaded, increasing the overall read rate of the cache pool.

At Etsy, they manually added cache smearing to hottest keys, with entropy in a small range like 0 to 8 or 0 to 16.

Understanding the Foundational Principles is the key to Effective Learning!

Follow along to Improve System Design Skills.

Reference and Image Credit

• How Etsy caches: hashing, Ketama, and cache smearing

Learn the tradeoffs for your next System Design Interview 👇

Definition

Vertical partitioning is when a database is split by moving tables and/or columns into a separate database

Background

Initially they were on a single large Amazon RDS database and observed upwards of 65% CPU utilization during peak traffic with Database latencies becoming increasingly unpredictable

First Tactical Fixes

1. Upgrade database to the largest instance available (from r5.12xlarge to r5.24xlarge) to maximize CPU utilization runway

2. Create multiple read replicas to scale read traffic

3. Establish new databases for new use cases to limit growth on original database

4. Add PgBouncer as a connection pooler to limit the impact of a growing number of connections

on further Analysis they learned

• Writes contributed a significant portion of database utilization.

• Not all reads could be moved to replicas due to application sensitivity to replication lag.

Hence, they still needed to offload more work from our original database.

Possible Solutions

1. Horizontal Scaling the Database

They observed many popular managed solutions are not natively compatible with Postgres.

So, they would either have to find a Postgres-compatible managed solution, or self-host.

2. Migrating to NoSQL

It would require

• Complex double read and write migration

• Significant application side changes

3. Postgres-compatible NewSQL

They would’ve had one of the largest single-cluster footprints for cloud-managed distributed Postgres.

So they didn’t want to bear the burden of being the first customer to hit certain scaling issues while having little control over managed solutions.

4. Self Hosting

It would mean

• Upfront work to acquire the training, knowledge, and skills to support self-hosting.

• Large operational cost

5. Vertically Partition the DB

They would move groups of tables onto their own databases.

This proved to have both short- and long-term benefits

1. Vertical partitioning relieves the original database

2. Provide a path forward for horizontally sharding subsets of tables in the future

How they implemented Vertical Partitioning

They first need to identify tables to partition into their own database. They chose the tables via following factors

1. Impact: Moving the tables should move a significant portion of workload

2. Isolation: The tables should not be strongly connected to other tables

1. Measuring impact

• They looked at Average Active Sessions (AAS) for queries (average number of active threads dedicated to a given query at a certain point in time).

• They calculated this information by querying pg_stat_activity in 10 millisecond intervals to identify CPU waits associated with a query, and then aggregated the information by table name.

2. Measuring isolation

• They have Ruby based backend which use ActiveRecord to write queries

• They created Validators that hooked into ActiveRecord and send production query and transaction information into Snowflake to look for queries and transactions that consistently referenced the same group of tables

• If these workloads turned out to be costly, those tables would be identified as prime candidates for vertical partitioning

Table Migration

The data movement has to be coordinated across thousands of application backend instances, so they could route queries to the new database at the correct moment.

They wanted a solution that met the following goals:

1. Limit potential availability impact to <1 minute

2. Automate the procedure so it is easily repeatable

3. Have the ability to undo a recent partition

Solution

At a high level, they implemented the following

1. Prepare client applications to query from multiple database partitions

2. Replicate tables from original database to a new database until replication lag is near 0

3. Pause activity on original database

4. Wait for databases to synchronize

5. Reroute query traffic to the new database

6. Resume activity

1. Preparing client application

They leveraged PgBouncer and created a separate PgBouncer services to virtually split traffic.

Partitioning the PgBouncer layer first would give clients leeway to route queries incorrectly.

They would be able to detect the routing mismatch, but since both PgBouncers have the same target database, the client would still successfully query data.

Start state looks as follows

State of the database after partitioning PgBouncer

State of the database after partitioning data

Once they verified that applications are prepared with separate connections for each PgBouncer (and sending traffic appropriately), they’d proceed.

2. Data Replication

There are two ways to replicate data: Streaming Replication or Logical Replication.

They chose logical replication because it allows them to

1. Port over a subset of tables, so they can start with a much smaller storage footprint in the destination database

2. Replicate to a database, which is running a different Postgres major version, meaning we can perform minimal-downtime major version upgrades with this tooling.

3. Set up reverse replication, which allows us to roll back the operation.

Working with terabytes of production data, Logical Replication was quite slow because it copies rows in bulk but inefficiently updates indexes one row at a time.

So they removed indexes in the destination database and rebuild the indexes after the initial copying reducing the copy time to a matter of hours.

Modifications to the new database were also replicated back to the old database (via reverse replication stream), for the case they rolled back.

3. Coordinating Query Rerouting

They performed sharding operation in two phases (partitioning PgBouncers, then data)

Overview of the operation:

• They stoped all relevant database traffic briefly in order for logical replication to synchronize the new database

• PgBouncer paused new connections and revoked clients’ query privileges on the partitioned tables in the original database.

• After a brief grace period, they canceled any remaining in flight queries.

• With traffic paused, they used LSNs (Log Sequence Number) to determine if two databases were synchronized. They sample an LSN from original database and waited for the replica to replay past this LSN. At this point, the data is identical in both the original and the replica.

A visualization of our synchronization mechanism

Post checking the replica is synchronized, they stoped replication and promote the replica to a new database.

Reverse replication is set up as previously mentioned. Then, they resume traffic in PgBouncer, but now the queries are routed to the new database.

A summary of the procedure

Conclusion

They observed a ~30 second period of partial availability impact (~2% of requests dropped).

Each database partition is operating with greatly increased headroom. Their largest partition has CPU utilization hovering ~10%

Understanding the Foundational Principles is the key to Effective Learning!

Follow along to Improve System Design Skills.

Reference and Image Credit

• How Figma Scaled to multiple databases

Cost saving of $120MM/year on EC2 on-demand R5 instance charges 🔥

Background

It started with migration from Oracle

• To decouple storage and compute, they migrated over 50 PB of data from Oracle to S3 (storage) + Redshift / RDS / Hive / EMR (for compute)

• Teams can now subscribe to S3-based table using their choice of analytics framework (e.g. Athena, Redshift, Spark, Flink etc) to run on-demand or scheduled queries.

• They further built data cataloging metadata layers, data discovery services, job orchestration infrastructure, web service APIs and UIs, and finally released the minimum viable set of services they needed to replace Oracle.

As all tables in their catalog composed of unbounded streams of S3 files, its the responsibility of each subscriber’s chosen compute framework to “merge,” all of the changes (inserts, updates, and deletes) at read time to yield the correct current table state.

Problem 1

With time, data has grown too large that "merge" operation started taking days or weeks to complete a merge, or they would just fail.

Solution

To solve this, BDT leveraged Spark on EMR to run the merge once and write back a read-optimized version of the table for other subscribers to use (copy-on-write)

They called this Spark job as Compactor

Problem 2

With time Amazon’s petabyte-scale data had grown to exabyte-scale.

Compaction jobs were exceeding their expected completion times, and had limited options to resolve performance issues as Spark abstracting away most of the low-level data processing details.

Solution

BDT Team did a PoC on Ray, with proper tuning they could compact 12X larger datasets than Spark, improve cost efficiency by 91%, and process 13X more data per hour

Factors that contributed to these results,

• Ray’s ability to reduce task orchestration and garbage collection overhead

• Leverage zero-copy intranode object exchange during locality-aware shuffles

• Better utilize cluster resources through fine-grained autoscaling.

• Flexibility of Ray’s programming model, which let them hand-craft a distributed application specifically optimized to run compaction as efficiently as possible.

They settled on an initial design for serverless job management using Ray on EC2 together with DynamoDB, SNS, SQS and S3 for durable job lifecycle tracking and management.

The solid lines of the primary workflow show the required steps needed to start and complete compaction jobs, while the dashed lines in the secondary workflow show the steps supporting job observability.

Migration of Spark Jobs to Ray

Initially they focused on migrating the tables that would yield the biggest impact first.

They targeted the largest ~1% of tables in their catalog, which accounted for ~40% of their overall Spark compaction cost and the vast majority of job failures.

Shadow Testing

They manually shadowed a subset of Spark compaction jobs on Ray by giving it the same input dataset as Spark.

They relied on their Ray-based data quality service to compare their respective outputs for equivalence.

Anything that didn’t pass DQ would be the target of a manual analysis, and either become a known/accepted difference to document or an issue to fix.

Lastly, they built a Data Reconciliation Service which compared the results of Spark and Ray-produced tables across multiple compute frameworks

Later they moved onto fully automated shadow compaction on Ray

This step required automatic 1:1 shadowing of all compaction jobs for any given table between Spark and Ray.

The purpose of this step was to get more direct comparisons with Spark, verify that Ray’s benefits held up across a wide variety of notoriously problematic tables, and to smoke out any latent corner-case issues that only appeared at scale.

To help avoid catastrophic failure, BDT built another service that let them dynamically switch individual table subscribers over from consuming the Apache Spark compactor’s output to Ray’s output.

Conclusion

BDT team read over 20PiB/day of S3 data to compact across more than 1600 Ray jobs/day.

Ray has maintained a 100% on-time delivery rate with 82% better cost efficiency than Spark per GiB of S3 data compacted, which translates to $120MM/year on EC2 on-demand R5 instance charges.

#systemdesign #architecture #scalability

Understanding the Foundational Principles is the key to Effective Learning!

Follow along to Improve System Design Skills.

Reference and Image Credits

• Amazon’s Exabyte-Scale Migration from Apache Spark to Ray on Amazon EC2

Use the proven techniques in your next System Design Interview 🔥

Background

Media metadata is classified as

1. Data on the post model e.g. video thumbnails, playback URLs, bitrates etc

2. Metadata associated with the lifecycle of the media asset, e.g. processing state, encoding information, S3 file location etc.

Challenges

1. Metadata distributed across multiple systems, resulting in inconsistent storage formats and varying query patterns.

2. Lack of proper mechanisms for auditing changes, analyzing content and categorizing metadata.

Functional Requirements

1. Move all existing media metadata into a Unified Storage.

2. Handle >100,000 read requests per second with a very low latency

3. Data creation and updates have lower traffic compared to Reads, slightly higher latency is tolerable

Solution

Reddit opted for AWS Aurora Postgres over Cassandra due to the challenges associated with ad-hoc queries for debugging in Cassandra, and the potential risk of some data not being denormalized and unsearchable.

1. Abstraction

Reads and writes to database are abstracted behind APIs.

2. Data Migration

Migration Process

1. Enabling dual writes into our metadata APIs from clients of media metadata.

2. Backfill data from older databases to our metadata store

3. Enable dual reads on media metadata from our service clients

4. Monitor data comparisons for each read and fix data gaps

5. Slowly ramp up the read traffic to our database to make sure it can scale

Scenarios leading to data differences

1. Data transformation bugs in the service layer.

2. Writes into the new metadata store failed, while writes into the source database succeed

3. Race condition when data from the backfill process in step 2 overwrites newer data from service writes in step

How to fix Data Differences

• Kafka consumer was set up to listen to a stream of data change events from the source database.

• The consumer then performs data validation with the media metadata store.

• If any data inconsistencies are detected, the consumer reports the differences to another data table in the database.

• This allows engineers to query and analyze the data issues.

3. Scaling Strategies

Media metadata was optimised for reads.

At 100k requests per second, read latency of 2.6 ms at p50, 4.7 ms at p90, and 17 ms at p99.

which is 50% faster than our previous data system

3a Table Partitioning

Size of media metadata will reach roughly 50 TB by the year 2030.

To address this they implemented table partitioning in Postgres using a partition management extension called pg_partman

And choose range-based partitioning (post_id as partition key) instead of hash-based partitioning.

As post_id increases monotonically with time, range-based partitioning allows to partition the table by distinct time periods.

This approach offers two important advantages

1. Most read operations are for recently created posts, allowing the Postgres engine to cache the indexes of the most recent partitions, minimizing disk I/O. Hot working set remains in memory, enhancing query performance.

2. Many batch read requests were on multiple post IDs from the same time period. Hence, likely the data is retrieved from a single partition.

3b. JSONB

Storing the media metadata as serialized JSONB format for Reads, effectively transformed the table into a NoSQL-like key-value pair.

Benefits

• It efficiently fetches all the fields together using a single key

• It eliminates the need for joins, simplifying the querying logic

Which other Database you will choose to optimise for Reads?

Understanding the Foundational Principles is the key to effective learning!

Follow along to Improve System Design Skills.

Reference and Image Credits

• https://www.reddit.com/r/RedditEng/comments/1avlywv/the_reddit_media_metadata_store/

Issues faced when Cassandra Cluster grew from 12 Nodes (in 2017) to 177 Nodes (in 2022)

Problems

In 2017, 12 Cassandra nodes were storing Billions of messages which grew to 177 nodes storing Trillions of messages.

It became a high-toil system, with the following issues

• Unpredictable Latencies

• The on-call team frequently gets paged for database issues

• Multiple Concurrent Reads and Uneven Discord Server sizes hot-spotting a partition, (given Read in Cassandra are more expensive than Writes) affecting latency across the entire cluster

• Compactions were falling behind creating cascading latency as a node tried to compact

• To avoid GC pauses (which would cause significant latency spikes) a large amount of time was spent tuning the JVM’s garbage collector and heap settings

Architecture Changes

Fixing Hot Partitions

• Enabling Request Coalescing (Query DB once for multiple requests for the same data) by building Data Services in Rust, as intermediary services that sit between API and database clusters containing roughly one gRPC endpoint per database query and no business logic

• Building efficient consistent hash-based routing to the data services which routes requests for the same channel to the same instance of the service

Migration to Scylla DB

• The data service library was extended to perform large-scale data migrations which read token ranges from a database, checkpoints them locally via SQLite, and then firehoses them into ScyllaDB

• The estimated time was 9 days with a speed of 3.2 million per second

• With an additional hiccup of intervention to compact gigantic ranges of tombstones (that were never compacted in Cassandra), migration was complete

• Automated data validation was performed by sending a small percentage of reads to both databases and comparing the results

Post Migration

• Decrease in Node count from 177 Cassandra Nodes to 72 ScyllaDB nodes

• Each ScyllaDB node has 9 TB of disk space, up from the average of 4 TB per Cassandra node

• Improved Tail latencies from p99 of between 40-125ms on Cassandra to 15ms p99 on ScyllaDB

• Improved message insert performance from 5-70ms p99 on Cassandra, to a steady 5ms p99 on ScyllaDB

Reference and Image Credit

• discord.com/blog/how-discord-stores-trillions-of-messages

They hosts >1,000,000,000,000,000,000 byte of data across tens of thousands of servers. 🔥

Uber, managing a mind-boggling 1 exabyte of data, has strategically migrated its Batch Analytics and ML training stack to Google Cloud Platform (GCP).

Here's a breakdown of their strategy.

Key Decisions

1. Adopted GCP's object store for storage

2. Migrate the Data Stack to GCP's IaaS (Infrastructure as a Service)

This 'lift and shift' approach allowed for a rapid migration with minimal disruption, as they could replicate their existing setup on GCP's IaaS.

Future plans include leveraging GCP's PaaS offerings (Dataproc, BigQuery) for even more elasticity and performance.

Migration Principles

1. Seamless Data Access for Users

• IaaS prevented users from needing to modify their artifacts or services.

• Cloud storage connector ensured HDFS compatibility.

• They standardized HDFS clients to abstract the on-prem HDFS implementation specifics. The standardized HDFS client will be modified to translate the HDFS paths to Object store based paths via a “Path Translation Service.”

2. Enhanced Data Access Proxies

Abstracted the underlying compute clusters for Presto, Spark, and Hive.

3. Cloud-Agnostic Infrastructure

Uber's container environment and deployment tools were already built to be cloud-agnostic, making the expansion to GCP smoother.

4. Proactive Data Governance

Focused on supporting selected, approved GCP data services to avoid future complexities.

Reference and Image Credit

• Modernizing-ubers-data-infrastructure-with-gcp

Are you tired of wrestling with Python project dependencies, virtual environments, and the complexities of packaging?

Let's revisit the challenges that often plague Python projects:

👉 Inconsistent Environments: Have you ever run into a "works on my machine" scenario? Mismatched dependencies can cause headaches when code behaves differently across environments.

👉 Version Conflicts: The dreaded dependency hell – when incompatible versions clash and break your code.

👉 Manual Tracking: Keeping track of every package and its version can quickly become overwhelming.

👉 Complex Packaging: Preparing your project for distribution can involve a series of manual steps.

Imagine a world where setting up your Python projects is as easy as pie.

Welcome to the world of Poetry!

Poetry addresses these issues head-on, by providing a streamlined and efficient way to handle dependencies, package projects etc making your Python projects more reliable and easier to manage.

In this beginner-friendly tutorial, we'll guide you through how to create Python projects using Poetry, step by step.

Everything you need to know.

Step 1: Installing Poetry (via Official Installer)

To embark on our Poetry journey, we need to install it.

The recommended way is via the official installer.

You can find detailed instructions on the Poetry installation page.

# For Linux, macOS, Windows (WSL)

curl -sSL https://install.python-poetry.org | python3 -

Step 2a: Creating Your First Python Project with Poetry

Once Poetry is installed, let's create a new Python project using Poetry's intuitive new command:

poetry new my-awesome-project

Project Structure

After running the command, Poetry generates a project structure that looks like this:

my-awesome-project/
├── pyproject.toml   
├── README.md 
├── my_awesome_project/
│   └── __init__.py
├── tests/
    └── __init__.py

• pyproject.toml: The heart of your project! This file contains project metadata, dependencies, and scripts.

• README.md: A place to describe your project in markdown format.

• my_awesome_project/: Your project's source code directory.

• tests/: A dedicated folder for your project's tests.

Step 2b: Breathing Poetry into Existing Projects

If you already have a Python project you want to manage with Poetry, it's easy to migrate.

Navigate to your project's root directory and run:

cd pre-existing-project
poetry init

Poetry will guide you through creating a pyproject.toml file, asking questions about your project's name, description, and dependencies.

Step 2c: Demystifying pyproject.toml

The pyproject.toml file generated via poetry new my-awesome-project command includes the following components:

[tool.poetry]
name = "my-awesome-project"
version = "0.1.0"
description = ""
authors = ["Kamran Ali <kamranalinitb@gmail.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.12"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

• [tool.poetry]: Contains the main metadata for your project such as name, version, description, authors.

• [tool.poetry.dependencies]: Lists all the dependencies required to run your project.

• [build-system]: Specifies the build system requirements, including requires and build-backend which are necessary for building your project.

Step 3: Poetry's Operating Modes

Poetry offers two operating modes:

• Package Mode: (default) Use this mode, if you want to package your project into an sdist or a wheel and publish it to a package index

  • Metadata such as name and version which are required for packaging, are mandatory.

  • The project itself will be installed in editable mode when running poetry install.

• Non-package Mode: Use this mode, If you want to use Poetry only for dependency management

We can disable package mode via

[tool.poetry]
package-mode = false

Step 4: Virtual Environments Made Easy with Poetry

Poetry manages virtual environments automatically.

By default, they are created in a central location

But you can configure Poetry to create virtual environments within your project directory by setting virtualenvs.in-project Config.

Having virtual environments within the project directory simplifies project management and ensures all project-related files are in one place.

Secondly, tools like vscode can detect the virtual environment of our project

# Enable virtualenvs.in-project via following command

poetry config virtualenvs.in-project true

# To list poetry configs
poetry config --list

Virtual Environment Commands

External Virtual Environment Management

Poetry can also work with external virtual environments. Just activate the external virtual environment before running Poetry commands.

Step 5: Mastering Dependency Management with Poetry

Dependencies are the building blocks of any Python project. Poetry makes adding, removing, and updating them a breeze.

Adding and Removing Dependencies

To add a dependency:

poetry add <package>

To remove a dependency:

poetry remove <package>

Showing and Updating Packages

To show installed packages:

poetry show

To update packages:

poetry update

# Update specific packages 
poetry update package_1 package_2

Poetry Dependency Management Commands

The Importance of poetry.lock

The poetry.lock file is your project's snapshot of dependencies and their exact versions.

It ensures that everyone working on your project has the same environment preventing surprises down the road.

Step 6: Dependency Groups

Poetry introduces dependency groups, a powerful feature for organizing your project's dependencies logically based on their purpose.

You can think of dependency groups as labels associated with your dependencies

Common groups include main and test

Creating a Test Group

poetry add --group <group> <package>

poetry add --group test pytest

The above command will create following section in pyproject.toml

[tool.poetry.group.test.dependencies]
pytest = "^8.2.2"

Creating Optional Groups

Optional groups let you define dependencies that are not always required

To create an optional group:

[tool.poetry.group.docs] # Marking "docs" group as Optional
optional = true

[tool.poetry.group.docs.dependencies]
mkdocs = "*"

Step 7: Packaging Your Project with Poetry

Ready to share your masterpiece with the world?

Poetry makes packaging a breeze

To build your project:

poetry build

This command creates both a source distribution (.tar.gz) and a wheel (.whl) in the dist/ directory, ready for installation.

To create only wheel, use following command

poetry build -f wheel

Step 8: Publishing to PyPI or Private Repositories

Poetry streamlines publishing your package to the Python Package Index (PyPI):

To publish your package:

poetry publish

To publish to a private repository, you need to configure the repository URL:

poetry config repositories.<repo-name> <repo-url>

Then publish using:

poetry publish -r <repo-name>

Conclusion

Python Poetry empowers you to create, manage, and share your Python projects with confidence.

Say goodbye to dependency nightmares and hello to a streamlined development workflow.

From installation to publication, Poetry's intuitive interface and powerful features make it an indispensable tool for beginners and experienced developers alike.

So, why not give Poetry a try for your next Python adventure? You might just find yourself wondering how you ever managed without it.

Are you ready to start your next Python project with Poetry?

Reference

• https://python-poetry.org/docs/basic-usage/

• https://python-poetry.org/docs/managing-dependencies/

• https://python-poetry.org/docs/libraries/

👉 You are facing frustrations with version conflicts between different projects requiring specific Python versions?

👉 You are concerned about environment pollution caused by globally installing various packages?

👉 You are struggling to replicate your development environment on another machine due to dependency issues?

👉 You are worried about the risk of breaking your system during package installation or uninstallation?

If you answered yes to any of these, this guide is here to help!

Let's Begin!

Pre Requisites

• MacBook

• Homebrew Installation

Step 1: Installing Homebrew (if not already installed)

Before we dive into Python installation, let's install Homebrew.

It's a package manager that makes installing software on your Mac a breeze.

Open your terminal and run the following command:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Step 2: Installing PyEnv

• This tool helps in installing and easily switching between multiple versions of Python.

• We can set Global Python version on a per-user basis.

• We can set Per Project Python versions.

2a Installing PyEnv

Run brew install pyenv to install PyEnv.

Post-installation, run pyenv --version to confirm the installation.

Next, we need to set up our shell environment for Pyenv.

For zsh Shell, run the following commands:

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
echo '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
echo 'eval "$(pyenv init -)"' >> ~/.zshrc

For other shells, refer this section

2b PyEnv Useful Commands

To install a version of Python

1. To do that, first, we need to check all the available versions of Python via pyenv install --list.

2. Run pyenv install <version> to install a particular version.

3. Example

   1. pyenv install 3.11.8

   2. pyenv install 3.10.13

Let's install both of them

If you see below error while installing 3.11.8

Traceback (most recent call last):

  File "<string>", line 1, in <module>In

  File "/Users/atech-guide/.pyenv/versions/3.11.8/lib/python3.11/lzma.py", line 27, in <module>
tppl
    from _lzma import *

ModuleNotFoundError: No module named '_lzma'

As solution, do the following

• Install xz => brew install xz

• un-install Python => pyenv uninstall 3.11.8

• reinstall Python => pyenv install 3.11.8

Post successful installation, let's list the Python versions via pyenv versions, we see the following:

atech.guide@macbook-air-m1 ~ % pyenv versions
* system (set by /Users/atech-guide/.pyenv/version)
  3.10.13
  3.11.8

Where system represents the Python that comes with MacBook.

To Use a version of Python in the current Folder

Run pyenv local <version>

To set a version of Python Globally

Run pyenv global <version>

E.g. pyenv global 3.10.13

To delete a version of Python

Run pyenv uninstall <version>

E.g. pyenv uninstall 3.10.13

2c PyEnv Example

Let's create a folder named python_projects .

Under it let's create two example projects, example_python_project_1 and example_python_project_2 .

At end, we have a folder structure as follows

atech.guide@macbook-air-m1 python_projects % tree
.
├── example_python_project_1
└── example_python_project_2

Under each project sub folder, lets use different versions of python by running

• pyenv local 3.11.8 for example_python_project_1

• pyenv local 3.10.13 for example_python_project_2

If we check the python versions under both the sub folders (via python --version) we will see different versions

atech.guide@macbook-air-m1 python_projects % cd example_python_project_1
atech.guide@macbook-air-m1 example_python_project_1 % pyenv local 3.11.8
atech.guide@macbook-air-m1 example_python_project_1 % python --version
Python 3.11.8

atech.guide@macbook-air-m1 example_python_project_1 % cd ../example_python_project_2

atech.guide@macbook-air-m1 example_python_project_2 % pyenv local 3.10.13      
atech.guide@macbook-air-m1 example_python_project_2 % python --version
Python 3.10.13

2d PyEnv Command Table

This solves one part of the problem.

Another part is we need to create Isolated Environments for each Python Project.

For that, we will need another tool.

Step 3 PyEnv Virtualenv

• It is a PyEnv plugin that provides features to manage virtualenvs for Python.

3a Installing PyEnv Virtualenv

Run brew install pyenv-virtualenv to install PyEnv

Next, we need to set up our shell environment for Pyenv Virtualenv.

For zsh Shell, run echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.zshrc.

3b PyEnv Virtualenv Useful Commands

To create a virtual env

Run pyenv virtualenv <python_version> <environment_name>.

To use virtual env

Run pyenv local <my-virtual-env>

To List virtual env

Run pyenv virtualenvs

To delete virtual env

Run pyenv virtualenv-delete <my-virtual-env>

E.g. pyenv virtualenv-delete python_project-3.11

3c PyEnv Virtualenv Example

Before Creating Virtual environment

atech.guide@macbook-air-m1 ~ % pyenv versions
* system (set by /Users/atech-guide/.pyenv/version)
  3.10.13
  3.11.8

Creating a Virtual Environment -> pyenv virtualenv 3.11.8 example_python_project_1

atech.guide@macbook-air-m1 ~ % pyenv virtualenv 3.11.8 example_python_project_1 
atech.guide@macbook-air-m1 ~ % pyenv versions
* system (set by /Users/atech-guide/.pyenv/version)
  3.10.13
  3.11.8
  3.11.8/envs/example_python_project_1
  example_python_project_1 --> /Users/atech-guide/.pyenv/versions/3.11.8/envs/example_python_project_1

Where 3.11.8/envs/example_python_project_1 is virtual env and example_python_project_1 is Symlink

3d PyEnv Virtualenv Command Table

Congratulations!

You've successfully installed and configured Python on your Mac using PyEnv and PyEnv Virtualenv.

You're now equipped to tackle Python projects with confidence, knowing you can manage versions and dependencies like a pro.

Conclusion

• PyEnv and PyEnv Virtualenv are powerful tools for managing Python development environment.

• Virtual environments keep our projects organized and prevent dependency conflicts.

Let me know if you'd like a follow-up article on setting up your first Python project.

References

• Homebrew

• PyEnv GitHub

• PyEnv pyenv-virtualenv GitHub

• Python 3.11.8 installation error solution Gist

• Image Credit: Google Gemini

• Reactive Systems
• Reactive Programming
• Functional Reactive Programming
• Reactive Design
• Reactive Architecture Patterns
• Reactive application

and may be much more ...

I would argue every topic requires an article for itself and I would like to primarily focus on Reactive Systems in this article.

Before proceeding forward I would like to bring us all on same page by defining few terms that I will frequently use in this article. Beginning with,

System

A system provides set of services to its clients/Users

For example, a website allowing us to book a Hotel is a System in itself. Keeping it simple, lets say it provides its users/clients with following services

• Making a reservation in the hotel of their choice
• Cancellation

A system is then further divided into Components. So let's define

Component

Components of a System collaborate with each other to provide services.

Continuing with the example of Booking System, Making a reservation is one service which require following components to talk to each other

• Front End: Website loaded in the browser
• User management: Maitaining profile of user
• Payment: Which charges the Credit/Debit Card
• Notification: Informing customer and user

... and much more you got the idea.

Interaction between these components can be simple client-server Or queue based and so on.

We can optinally group few components and call them Sub Systems.

You know the drill.

Moving on lets begin by answering a simple question,

What are Reactive Systems?

I personally like the Reactive Systems definition on Oreilly blog, which says

A reactive system is an architectural style that allows multiple individual applications to coalesce as a single unit, reacting to its surroundings, while remaining aware of each other.

If above definition looks too abstract, lets try again

Reactive Systems in a nutshell is an Architectural and Design pattern of building large scale, responsive, resilient, self healing systems where individual components talk to each other over Asynchronous Messaging.

I guess above definition packs too many "buzzword". Trust me I will add details. :)

First lets focus on

Why Reactive System

As rightly mentioned in Reactive manifesto,

Today's demands are simply not met by yesterday’s software architectures

Applications developed today requires to have

• Faster release cycle to ship new functionality quickly
• Milliseconds response time
• Cloud based clusters
• 100% uptime
• Tolerant to failure

and much more depending on the use cases.

Even the underlying hardware has evolved to multi-core processors generating compute capacity to handle data in petabytes.

The requirements mentioned above are more or less common in every organisation trying to solve a problem through technology.

Even though the problems may be completely different but the guidelines and design principles that we choose to solve it, does overlap.

Shall we benefit from those already tested Solutions?

Definitely, Yes !!

Like-minded people have already identified common patters in those problems and drafted guidelines which may act as blue print for next generation application.

One such attempt is Reactive Manifesto, which believes

All necessary aspects are already recognised individually

Basically it says,

Systems should be Responsive, Resilient, Elastic and Message Driven.

They call such systems as Reactive Systems.

So without further ado, lets drill into the

Characteristics of Reactive System

Responsiveness

Responsiveness means that Reactive Systems

• Should respond in consistent and timely manner.
  • when an API call is made we expect a predictable and consistent response within SLAs. It helps in setting a reliable upper bound paving way for effective error handling and building confidence for its clients.
• Will detect problems quickly and mitigate it effectively.
  • It demonstrates the self healing nature of reactive systems in which once bleeding begins, it is detected quickly and system takes appropriate actions to mitigate it.

Resilient

Resilience means that Reactive Systems should be responsive in face of failure.

i.e. Failure(s) in one component should neither burden its client for handling the failures nor should bring the entire system to a grinding halt.

Resilience is achieved by

• Replication: To ensure high availability we use replication. e.g. replicating master nodes of Redis Cluster.
• Containment: Failures are contained within each component. e.g. Failures in component responsible for lets say Async Notification should not bring down Sync Notification.
• Isolation: Components of a system are isolated ensuring part of system can fail and recover without compromising the entire system.
• Delegation: Recovery of each component is delegated to another component (maybe external). e.g. falling back to alternate data source if primary data source is unavailable

Elastic

Elasticity means that Reactive Systems should be responsive under varying workload.

Reactive Systems react to changes in the input rate by increasing or decreasing the resources allocated to service these inputs. Best example to understand this situation is cluster of similar services behind a load balancer. When load increases additional nodes are added via predictive or reactive scaling algorithm and vice versa.

Management of resources under varying workload is a very powerful concept

• Ensuring we have no contention points or central bottlenecks
• Enabling us to shard or replicate components and distribute inputs among them

Note: Elasticity is achieved in cost-effective way on commodity hardware and software platforms.

Message Driven

Reactive systems have Asynchronous Messaging i.e. while requesting data we don’t wait for response, instead we register a callback and return. When data is available, it is pushed to callback method.

Asynchronous Messaging establishes a boundary between components which further helps in ensuring

• Loose coupling and Isolation implying sender and receiver can have independent life-cycles. They do not need to be present at the same time and run on same process for communication to be possible. For example, queue based systems in which sender is pushing into queue and reciever is pulling from the queue.
• Elasticity implying throughput of a system scales up or down automatically to meet varying demand.
• Location transparency (also known as Decoupling in space) implying we are not dependent on server A talking to server B instead a cluster of micro service 'A'talks to cluster of micro service 'B'.
• Load Management implying sender and receiver can scale independently. For example, if the rate by which sender is pushing messages in the queue increases then reciever can scale independently to consume and process messages from the queue.

Note

• Boundary between components ensures failures are notified in form of messages.
• Non-blocking communication facilitates only consume resources while active, leading to less system overhead.

Conclusion

I would like to conclude it by suggesting the readers to look into/apply the characteristics discussed above, next time they are building New System. You may also look into the case study of How Paypal is handling billions of requests per day on 8 VMs and 2 vCPU? by building Reactive Systems on Akka and Squbs.

See everything boils down to the satisfaction of your customer/client when they visit your site. If site is slow, unresponsive and in worse case down, then that's terrible experience.

Take all steps to avoid it

Reference

• Reactive manifesto
• Reactive manifesto glossary
• Tech Primers - What is Reactive Programming?
• O Reilly - reactive-programming-vs-reactive-systems