Topics 11
Last updated
Last updated
How do we interact with Linux Filesystem via file descriptors?
A file descriptor represents an open file. It is a unique number assigned by the operating system to each file. It is an abstraction for working with files. We need to use file descriptors to read from or write to files in our program. Each process maintains its own file descriptor table.
The diagram below shows the layered architecture in Linux filesystem. Let’s take process 1234 as an example.
🔹 In User Space When we open a file called “fileA.txt” in Process 1234, we get file descriptor fd1, which is equal to 3. We can then pass the file descriptor to other functions to write data to the file.
🔹 In Kernel Space In Linux kernel, there is a process table to maintain the data for the processes. Each process has an entry in the table. Each process maintains a file descriptor table, with file descriptors as its indices. Notice that file descriptors 0,1 and 2 are reserved in each file descriptor table to represent stdin, stdout, and stderr.
The file pointer points to an entry in the open file table, which has information about open files across all processes. Multiple file descriptors can point to the same file table entry. For example, file descriptor 0, 1 and 2 points to the same open file table entry.
Since different open file table entries can represent the same file, it is a waste of resources to store the file static information so many times. We need another abstraction layer called ‘vnode table’ to store the static data.
In each file table entry, there is a vnode pointer, which points to an entry in vnode table. The static information includes file type, function pointers, reference counts, inode etc. inode describes a physical object in the filesystem.
🔹 In Filesystem The inode array element stores the actual file information, including permission mode, owners, timestamps, etc. inode also points to the data blocks stored in the filesystem.
Over to you: When we close a file in a program, do you know which entries are deleted in these data structures?
Do you know how you get paid at work? In the US, tech companies usually run payrolls via Automatic Clearing House (ACH).
ACH handles retail transactions and is part of American retail payment systems. It processes transactions in batches, not in real-time. The diagram below shows how ACH direct deposit works with payrolls.
🔹Step 0: Before we can use ACH network, the originator who starts the transactions needs to open an account at a commercial bank because only banks are allowed to initiate ACH transactions directly. The bank is called ODFI (Originating Depository Financial Institution). Then the transaction receiver needs to authorize the originator for certain types of transactions.
🔹Step 1: The originator company originates salary payment transactions. The transactions are sent to a 3rd-party processor like Gusto. The third-party processor helps with ACH-related services like generating ACH files, etc.
🔹Step 2: The third-party processor generates ACH files on behalf of the originator. The files are uploaded to an SFTP established by the ODFI. This should be done by the 7 PM cut-off time, as specified by the ODFI bank.
🔹Step 3: After normal business hours in the evening, the ODFI bank forwards the ACH files to the ACH operator for clearing and settlement. There are two ACH operators, one is the Federal Reserve (FedACH), and the other is EPN (Electronic Payment Network – which is operated by a private company).
🔹Step 4: The ACH files are processed around midnight and made available to the receiving bank RDFI (Receiving Depository Financial Institution.)
🔹Step 5: The RDFI operates on the receiver’s bank accounts based on the instructions in the ACH files. In our case, the receiver receives $100 from the originator. This is done when the RDFI opens for business at 6 AM the next day.
ACH is a next-day settlement system. It means transactions sent out by 7 PM on one day will arrive the following morning.
Since 2018, it’s possible to choose Same Day ACH so funds can be transferred on the same business day.
Over to you: ACH is a US financial network. If you live outside the US, do you know what payment method your employer uses to send payment? What’s the difference between ACH and wire transfer?
How do we build a simple chat application using Redis?
The diagram below shows how we can leverage the pub-sub functionality of Redis to develop a chat application.
🔹Stage 1: Connection Initialization
Steps 1 and 2: Bob opens the chat application. A web socket is established between the client and the server.
Steps 3 and 4: The pub-sub server establishes several connections to Redis. One connection is used to update the Redis data models and publish messages to a topic. Other connections are used to subscribe and listen to updates for topics.
Steps 5 and 6: Bob’s client application requires the chat member list and the historical message list. The information is retrieved from Redis and sent to the client application.
Steps 7 and 8: Since Bob is a new member joining the chat application, a message is published to the “member_add” topic, and as a result, other participants of the chat application can see Bob.
🔹Stage 2: Message Handling
Step 1: Bob sends a message to Alice in the chat application.
Step 2: The new chat message is added to Redis SortedSet by calling ‘zadd.’ The chat messages are sorted based on arrival time. The pub-sub server then publishes the chat message to the “messages” topic so subscribers can pick it up.
Step 3: Alice’s client application receives the chat message from Bob.
👉 Over to you: What backend stack is commonly used to build a large-scale chat application?
Airbnb’s microservice architecture went through 3 main stages. This post is based on the tech talk by Jessica Tai. See the reference link at the end of the thread.
Monolith (2008 - 2017)
Airbnb began as a simple marketplace for hosts and guests. This is built in a Ruby on Rails application - the monolith.
What’s the challenge?
Confusing team ownership + unowned code
Slow deployment
Microservices (2017 - 2020)
Microservice aims to solve those challenges. In the microservice architecture, key services include:
Data fetching service
Business logic data service
Write workflow service
UI aggregation service
Each service had one owning team
What’s the challenge?
Hundreds of services and dependencies were difficult for humans to manage.
Micro + macroservices (2020 - present)
This is what Airbnb is working on now. The micro and macroservice hybrid model focuses on the unification of APIs.
Over to you - why do you think both Airbnb and Netflix use GraphQL?
How does blockchain change the design of digital wallets? Why do VISA and PayPal invest in blockchains?
The diagram below explains the differences.
In banking systems
🔹Deposit process: Bob goes to Bank of America (BoA) to open an account and deposit $100. A new account B1234 is created in the wallet system for Bob. The cash goes to the bank’s vault and Bob’s wallet now has $100. If Bob wants to use the banking services of Citibank (Citi,) he needs to go through the same process all over again.
🔹Transfer process: Bob opens BoA’s App and transfers $50 to Alice’s account at Citi. The amount is deducted from Bob’s account B1234 and credited to Alice’s account C512. The actual movement of cash doesn’t happen instantly. It happens after BoA and Citi settle all transactions at end-of-day.
🔹Withdrawal process: Bob withdraws his remaining $50 from account B1234. The amount is deducted from B1234, and Bob gets the cash.
With Blockchains
🔹Deposit & Withdraw: Blockchains support cryptocurrencies, with no cash involved. Bob needs to generate an address as the transfer recipient and store the private key in a crypto wallet like Metamask. Then Bob can receive cryptocurrencies.
🔹Transfer: Bob opens Metamask and enters Alice’s address, and sends it 2 ETHs. Then Bob signs the transaction to authorize the transfer with the private key. When this transaction is confirmed on blockchains, Bob’s address has 8 ETHs and Alice’s address has 101 ETHs.
👉 Can you spot the differences?
Blockchain is distributed ledger. It provides a unified interface to handle the common operations we perform on wallets. Instead of opening multiple accounts with different banks, we just need to open a single account on blockchains, which is the address.
All transfers are confirmed on blockchains in pseudo real-time, saving us from waiting until end-of-day reconciliations.
With blockchains, we can merge wallet services from different banks into one global service.
How do video live streamings work on YouTube, TikTok live, or Twitch? The technique is called live streaming.
Livestreaming differs from regular streaming because the video content is sent via the internet in real-time, usually with a latency of just a few seconds.
The diagram below explains what happens behind the scenes to make this possible.
Step 1: The raw video data is captured by a microphone and camera. The data is sent to the server side.
Step 2: The video data is compressed and encoded. For example, the compressing algorithm separates the background and other video elements. After compression, the video is encoded to standards such as H.264. The size of the video data is much smaller after this step.
Step 3: The encoded data is divided into smaller segments, usually seconds in length, so it takes much less time to download or stream.
Step 4: The segmented data is sent to the streaming server. The streaming server needs to support different devices and network conditions. This is called ‘Adaptive Bitrate Streaming.’ This means we need to produce multiple files at different bitrates in steps 2 and 3.
Step 5: The live streaming data is pushed to edge servers supported by CDN (Content Delivery Network.) Millions of viewers can watch the video from an edge server nearby. CDN significantly lowers data transmission latency.
Step 6: The viewers’ devices decode and decompress the video data and play the video in a video player.
Steps 7 and 8: If the video needs to be stored for replay, the encoded data is sent to a storage server, and viewers can request a replay from it later.
Standard protocols for live streaming include:
RTMP (Real-Time Messaging Protocol): This was originally developed by Macromedia to transmit data between a Flash player and a server. Now it is used for streaming video data over the internet. Note that video conferencing applications like Skype use RTC (Real-Time Communication) protocol for lower latency.
HLS (HTTP Live Streaming): It requires the H.264 or H.265 encoding. Apple devices accept only HLS format.
DASH (Dynamic Adaptive Streaming over HTTP): DASH does not support Apple devices. Both HLS and DASH support adaptive bitrate streaming.
Over to you: What are some of the optimizations that can be done in this process? Which type of storage is suitable for video persistence in step 7
What are the differences between VISA and American Express’s (AMEX) processing when you swipe credit cards?
The major difference is VISA uses a 4-party model where the issuer and acquirer are different entities, while AMEX uses a 3-party model where the issuer and acquirer are the same entity.
The diagram below uses the authorization flow to demonstrate the differences.
🔹4-Party Model (Authorization Flow)
Step 0: The card-issuing bank issues a credit card to its customer.
Step 1: The cardholder buys a product by swiping their credit card at the Point of Sale (POS) terminal in a merchant’s shop.
Step 2: The POS terminal sends the transaction to the acquiring bank, which provides the POS terminal.
Steps 3 and 4: The acquiring bank sends the transaction to the card network, also called the Card Scheme. This card network sends the transaction to the issuing bank for approval.
Steps 4.1, 4.2, and 4.3: The issuing bank freezes the money if the transaction is approved. The approval or rejection is sent back to the acquirer, and to the POS terminal.
🔹3-Party Model (Authorization Flow)
Steps 0,1 and 2 are the same as in the 4-party model.
Step 3: Since one company performs issuing, acquiring, and card network functions, the transactions are processed internally within the franchisor. This is also called the closed loop card model. Closed loop networks are more efficient because all functions are processed in one franchisor. However, it doesn’t allow other entities to issue or acquire on its behalf, so it scales more slowly.
In recent years, the closed loop networks have partnered with other issuers and acquirers to scale their circulation.
Step 4: The approval or rejection is sent back to the acquirer, then to the POS terminal.
Why is the credit card called “the most profitable product in banks”? How does VISA/Mastercard make money?
The diagram below shows the economics of the credit card payment flow.
The cardholder pays a merchant $100 to buy a product.
The merchant benefits from the use of the credit card with higher sales volume, and needs to compensate the issuer and the card network for providing the payment service. The acquiring bank sets a fee with the merchant, called the “merchant discount fee”
3 - 4. The acquiring bank keeps $0.25 as the acquiring markup, and $1.75 is paid to the issuing bank as the interchange fee. The merchant discount fee should cover the interchange fee.
The interchange fee is set by the card network because it is less efficient for each issuing bank to negotiate fees with each merchant.
The card network sets up the network assessments and fees with each bank, which pays the card network for its services every month. For example, VISA charges a 0.11% assessment, plus a $0.0195 usage fee, for every swipe.
The cardholder pays the issuing bank for its services.
Why should the issuing bank be compensated?
The issuer pays the merchant even if the cardholder fails to pay the issuer.
The issuer pays the merchant before the cardholder pays the issuer.
The issuer has other operating costs, including managing customer accounts, providing statements, fraud detection, risk management, clearing & settlement, etc.
Over to you: Does the card network charge the same interchange fee for big merchants as for small merchants?
