Table of contents
Open Table of contents
- Introduction
- 1. Identity Unification Using Disjoint Set Union (DSU)
- 2. High-Throughput Background Job System (40K Jobs/Minute)
- 3. Deadlock-Resilient Transaction Processing
- 4. Personalized Notification System (20x CTR Improvement)
- 5. World’s First Polymarket Integration
- 6. Cross-Platform Social Graph Infrastructure
- 7. Optimized Onboarding Experience
- 8. Scheduled & Draft Posts Infrastructure
- 9. Enhanced People Recommendation System
- 10. Engagement API Optimization (20x Performance Boost)
- 11. Feed Engagement Metrics & Viral Content Detection
- 12. UserDeviceTracker System (Breaking Changes Without Breaking)
- Lessons Learned
- Technical Highlights
- Conclusion
Introduction
From February 2024 to November 2024, I had the incredible opportunity to work as an Early Engineer (OG Team) at 0xPPL, a social media platform founded by a Rippling co-founder. I was part of one of India’s top engineering teams, composed of 7 ICPC World Finalists (including 3 India Rank-1s).
During my time there, I became the Top 4 Backend Contributor with 600+ PRs merged in 11 months and had full ownership of multiple large-surface area projects that significantly impacted user engagement, system performance, and overall product experience.
This post is a comprehensive overview of the major systems I built and the technical challenges I solved.
1. Identity Unification Using Disjoint Set Union (DSU)
The Challenge
Users on 0xPPL connected from multiple social platforms (Twitter, Farcaster, Lens) and crypto wallets. However, the same person often had fragmented profiles across these platforms, making it difficult to:
- Show users their complete social graph
- Provide accurate friend recommendations
- Deliver personalized content feeds
The Solution
I designed and implemented a user identity resolution system leveraging Disjoint Set Union (DSU) algorithms to unify fragmented user profiles.
Key Features:
- Automatically unified profiles across Twitter, Farcaster, Lens, and wallet addresses
- Pre-generated 60-70 new friend identities per user during onboarding
- Enabled users to see complete profiles of their friends even before those friends had onboarded
Impact:
- Consolidated over 10,000 missing social account details
- Enhanced friend recommendation accuracy by 35%
- Injected 1,000-2,000 new unified identities weekly (rate stabilized due to saturation)
- Significantly improved personalized feed relevance
Technical Deep Dive
The DSU algorithm efficiently handles the “friend of a friend” relationships across platforms. When we detect that two accounts belong to the same person (via shared wallet addresses, ENS names, or other signals), we merge them in the DSU structure.
This allows O(α(n)) time complexity for union and find operations, where α is the inverse Ackermann function, making it nearly constant time in practice.
2. High-Throughput Background Job System (40K Jobs/Minute)
The Challenge
Our background job processing system was handling around 600-700 jobs per minute, but processing times were taking 30+ minutes. We needed to scale this dramatically for features like notification generation, social graph updates, and data aggregation.
The Solution
I architected an Auto-Scalable Recursive Binary Splitting Range Scheduler that could handle 30,000-40,000 jobs per minute — a 60x performance improvement.
Key Features:
- Recursive binary splitting of job ranges for optimal parallelization
- Live system health checks with auto-scaling capabilities
- Intelligent tree pruning that skips subtrees when ranges don’t need processing
- Dynamic expansion and shrinking based on workload
Impact:
- Reduced processing time from 30 minutes to under 1 minute
- Enabled near real-time background task processing
- Logarithmic scaling (O(log N)) for handling workloads of any size
Architecture
The scheduler works by taking a range of work items (e.g., user IDs 1-100000) and recursively splitting it into smaller ranges. Each worker processes a range, and if the range is still too large or the worker is overloaded, it further splits the range and delegates to other workers.
Initial: [1 - 100000]
Split: [1 - 50000] [50001 - 100000]
Split: [1 - 25000] [25001 - 50000] [50001 - 75000] [75001 - 100000]
...and so on
The system monitors health metrics and can prune branches early if certain conditions are met, avoiding unnecessary work.
3. Deadlock-Resilient Transaction Processing
The Challenge
Our system was experiencing frequent deadlocks during bulk database operations, especially when creating or updating large batches of records. These deadlocks required manual intervention and caused system instability.
The Solution
I designed a deadlock-aware bulk create/update transaction wrapper that automatically detects and resolves deadlocks.
Impact:
- Reduced deadlock failures by 96%
- 96 out of 100 deadlocks now auto-resolve without manual intervention
- Significantly enhanced system resilience and stability
How It Works
The wrapper detects deadlock exceptions from PostgreSQL and automatically retries the transaction with:
- Exponential backoff
- Record sorting to ensure consistent lock ordering
- Transaction isolation level optimization
- Batch size adjustment based on contention
4. Personalized Notification System (20x CTR Improvement)
The Challenge
Our notification system had a 0.5% click-through rate and was actually driving users away from the platform. Notifications were generic, poorly timed, and often irrelevant.
The Solution
I led a comprehensive overhaul of the notification system, transforming it into a highly personalized, contextual experience.
Key Features:
- Affinity-based filtering using user interaction history
- Dynamic frequency adjustment based on user behavior
- Time-aware notifications respecting user timezones
- 10x speed improvement through time-based pagination and dynamic windowing
- Expanded notification categories (profit alerts, top traders, social interactions)
Impact:
- 20x increase in click-through rates: from 0.5% to 6-7%
- Virtually eliminated notification-driven churn
- Reduced database load for notification fetching
- Fast access for all user segments (both sparse and dense notification users)
Technical Highlights
I implemented a sophisticated scoring system that considers:
- Recency of interaction
- Type of notification
- Historical engagement patterns
- User’s active hours
- Social graph proximity
The pagination system uses time-based windows rather than offset-based pagination, making it efficient even for users with thousands of notifications.
5. World’s First Polymarket Integration
The Challenge
Polymarket is the world’s largest decentralized betting market, operating across multiple blockchains. No other app had successfully integrated it. The challenge was:
- No formal documentation
- Complex proxy wallet system
- Encrypted blockchain data
- Complex collateral token mechanics
The Solution
I achieved the first-ever successful web3 integration of Polymarket in 3.5 weeks through reverse engineering.
Key Achievements:
- Decoded event signatures and parsed blockchain-encoded hex data
- Mapped internal logic of maker-taker behavior and order matching
- Built system to decrypt and present data in under 200 seconds (block-to-UI)
- Developed 3-level deep search by username to find linked wallets
- Beat Polymarket’s native UI in both speed and visual quality
Impact:
- Provided users with unique search capabilities not available on Polymarket itself
- Real-time portfolio tracking faster than the native platform
- Enhanced user engagement through timely bet notifications
6. Cross-Platform Social Graph Infrastructure
The Challenge
Building a social media platform that aggregates data from Twitter, Lens, and Farcaster required querying relationship data across billions of nodes in real-time.
The Solution
I architected a high-performance multiplatform infrastructure that manages social connection data at massive scale.
Key Features:
- Sub-100ms average query times for relationship queries
- Partial indexing leveraging the sparse nature of social graphs
- Dynamic cache policies: weekly refresh for scraped data, live updates for platform data
- Smart background refresh mechanism with optimal scheduling (max one refresh per user per hour)
Impact:
- Enabled real-time “who knows who” queries across platforms
- Massively improved recommendation algorithms
- Enhanced new user onboarding flow with instant social context
- Reduced server load through intelligent caching
Graph Query Optimization
The key insight was that social graphs are extremely sparse — most people are connected to a tiny fraction of all users. By using partial indexes on the most active users and connection types, we achieved:
- 98% query coverage with indexes covering only 15% of data
- Fallback to full scan only for edge cases
- Memory-efficient through selective materialization
7. Optimized Onboarding Experience
The Challenge
New users were experiencing a 5-minute wait during onboarding while we generated their personalized feed. This led to significant drop-off.
The Solution
I re-engineered the entire data processing flow to deliver personalized content instantly.
Key Achievements:
- Reduced feed generation time from 5 minutes to under 0.5 seconds (p95)
- Delivered 200-300 curated items in under 0.5 seconds (p75)
- Reduced user drop-off by over 12%
Impact:
- Users were amazed to see their friends’ activities instantly — even before following them
- Created a “wow” moment during onboarding
- Significantly improved first-time user experience and retention
Optimization Techniques
- Pre-computation: Calculated likely connections based on wallet addresses before user completes signup
- Parallel fetching: Queried multiple platforms simultaneously
- Smart caching: Cached common friend graphs
- Progressive loading: Showed initial results immediately while loading more in background
8. Scheduled & Draft Posts Infrastructure
The Challenge
Users wanted the ability to schedule posts and save drafts across multiple platforms (Twitter, Farcaster, Lens), but this required:
- Reliable timing mechanisms
- Cross-platform API orchestration
- Failure handling and retries
- Draft persistence and recovery
The Solution
I architected a highly-reliable system for scheduled and draft posts with cross-platform capabilities.
Key Features:
- Centralized scheduling service with distributed workers
- Per-platform adapters for API differences
- Automatic retry with exponential backoff
- Draft auto-save every 30 seconds
- Scheduled post verification before publishing
Impact:
- Became a key premium offering
- Supported seamless scheduling across multiple networks
- Enhanced content creator experience
9. Enhanced People Recommendation System
The Challenge
Recommending relevant people to follow required understanding connections across multiple platforms simultaneously.
The Solution
I developed an algorithm for people recommendations that considers multi-platform social intersections.
Key Features:
- Weighted scoring based on platform importance
- “Friend of friend” analysis across platforms
- Activity-based relevance scoring
- De-duplication of cross-platform profiles
Impact:
- Improved follow recommendations quality
- Enhanced social discovery
- Increased network growth rate
10. Engagement API Optimization (20x Performance Boost)
The Challenge
Critical engagement actions (likes, posts, replies, follows) had p75 latency of 2.5 seconds, making the app feel sluggish.
The Solution
I optimized the entire engagement pipeline through database query optimization, caching, and workflow improvements.
Impact:
- Reduced p75 latency to 0.2ms from 2.5s
- 20x performance boost on certain response flows (reply, repost)
- Significantly improved user interaction experience
Optimization Strategies
- Query optimization: Reduced N+1 queries through eager loading
- Redis caching: Cached user engagement state
- Async processing: Moved non-critical operations to background
- Database indexes: Added covering indexes for common queries
- Connection pooling: Optimized database connection management
11. Feed Engagement Metrics & Viral Content Detection
The Challenge
Detecting viral content and understanding engagement patterns in a heterogeneous social media graph (spanning multiple platforms) was an open-ended problem with no established solution.
The Solution
I developed an optimized algorithm to detect engagement loops using advanced graph traversal techniques.
Key Features:
- Mathematically defined engagement loops in multi-platform context
- Early pruning strategies to reduce computational overhead
- Tracking metrics like “followers before interaction”
- Distribution analysis of interactions (posts vs on-chain activities)
- Blockchain-to-notification latency tracking
Impact:
- Reduced computational overhead by 40%
- Enabled identification of viral content and influential interactions
- Provided insights into cross-platform engagement patterns
- Helped optimize transaction processing across different blockchains
12. UserDeviceTracker System (Breaking Changes Without Breaking)
The Challenge
Rolling out breaking changes (non-backward compatible features) while maintaining app stability for users on older versions.
The Solution
I developed a UserDeviceTracker system that manages feature rollouts based on app version and OS version.
Key Features:
- Device fingerprinting and version tracking
- Feature flag system with version constraints
- Gradual rollout capabilities
- Automatic fallback for incompatible devices
Impact:
- Enabled safe deployment of breaking changes
- Prevented app crashes for users on older versions
- Smooth upgrade experience for users
Lessons Learned
1. Start Simple, Scale Smart
Many of my systems started with simple implementations and evolved based on real usage patterns. The job scheduler, for instance, initially had fixed parallelism before I added auto-scaling.
2. Measure Everything
Having detailed metrics was crucial for optimization. I could only achieve 20x performance improvements because I knew exactly where the bottlenecks were.
3. User Experience is Technical
The “wow” moment during onboarding wasn’t just good UX design — it was the result of aggressive technical optimization that made instantaneous results possible.
4. Algorithms Matter
Competitive programming experience directly translated to real-world impact. DSU for identity resolution, graph algorithms for engagement detection, and binary splitting for job scheduling all came from algorithmic thinking.
5. Ownership Drives Impact
Having full ownership of large surface areas meant I could make holistic decisions rather than point solutions. This led to more impactful improvements.
Technical Highlights
- Languages: Go, Python, TypeScript
- Databases: PostgreSQL, Redis, MongoDB
- Scale: Billions of graph nodes, 40K jobs/minute, sub-100ms queries
- Impact Metrics: 20x CTR, 60x throughput, 96% deadlock resolution, 35% recommendation accuracy
Conclusion
My time at 0xPPL was incredibly rewarding. Being part of an elite engineering team pushed me to deliver my best work, and having ownership of critical systems taught me how to think about scalability, user experience, and system reliability at every level.
The systems I built continue to serve thousands of users, and the lessons I learned will guide my engineering decisions for years to come.
Currently, I’m bringing these learnings to my role as a Founding Engineer at Share.xyz, where I’m building collaborative tools from the ground up.
If you’re working on challenging problems in backend systems, distributed systems, or algorithms, I’d love to connect! Reach out at [email protected].
This post was last updated on November 10, 2024.