Networking 101: An Introduction for Programmers

One Program Wants to Talk to Another

Most introductions to networking start with a pile of terms: IP, TCP, UDP, DNS, HTTP, TLS, NAT, sockets. I find that hard to learn from. Before I can remember an answer, I need to understand the problem that made someone invent it.

So let’s start with something familiar:

curl https://example.com/index.html

curl is a command-line tool for transferring data. Here we are asking it to fetch a resource from a website. A browser does something similar when you enter a URL, then does more work to display the result.

From our program’s perspective, the task looks simple:

Our program  <-->  Web server

But what does example.com mean to a computer? How do bytes reach it? When they arrive, which program receives them, and how does that program know what we want? Can we trust the answer? What happens if something along the way fails?

These questions give us a path through the subject:

You only need basic programming knowledge to follow along. The shell examples use macOS, Linux, or WSL with a Unix shell. Some experiments need dig, nslookup, lsof, traceroute, or nc (netcat), which may need installing through your system’s package manager. For example, Debian and Ubuntu provide dig and nslookup in dnsutils. The local examples also use Python 3.

For a visual overview, you can also download my Network Principles for Programmers slides (PDF, 100 pages, 2 MB).

A Network of Networks

The Internet connects networks run by different organizations: homes, universities, companies, Internet service providers, and more. No single operator manages the entire path between your laptop and a remote server. This is what it means to call the Internet federated.

Those networks can also use different technologies. Your laptop might begin with Wi-Fi, while the next part of the journey uses Ethernet or an optical link. We need a common way to carry data across all of them. That is the role of IP, the Internet Protocol.

The design gives us two more useful ideas:

• End-to-end: functions that require knowledge of the whole conversation, such as checking that an application operation succeeded, belong at the endpoints. Intermediate devices help move traffic, but cannot establish every guarantee the application needs.
• Best effort: IP does not promise delivery, order, freedom from duplicates, or a fixed delay. Higher layers add the behavior they need.

What Layers Are For

You may have seen the seven-layer OSI model. Its practical value is separating responsibilities. For this article, a smaller map is enough:

OSI also names session and presentation layers. In everyday application development, the corresponding work often lives in libraries and application protocols: tracking a login session, encoding data as JSON, or negotiating encryption. These are useful responsibilities to recognize, but real software does not always divide neatly into seven boxes.

The layering lets a web application work across many kinds of networks. A router can forward IP packets without implementing your application’s API. We will return to the layers whenever they explain a concrete decision.

Who Do I Want to Talk To?

Our URL contains several pieces of information:

https://example.com/index.html
  |          |          |
 scheme    hostname    path

The scheme selects HTTPS, whose default port is 443. The hostname names the service. The path identifies a resource at that service. Each will matter at a different point in the conversation.

DNS: From a Name to an Address

A domain name can stay useful while the machines behind it change. One name can have several addresses, and one address can serve several names. That is why a name and an address are separate things.

DNS, the Domain Name System, lets programs ask for records associated with a name:

For a web connection, address records are the immediate concern. Try:

nslookup example.com
dig example.com A
dig example.com AAAA

Look at the answer section and the server that answered. The result may contain several addresses; it may differ across networks or over time. An address shown in a tutorial is not a permanent property of that domain.

Who Answers the DNS Question?

Usually your machine asks a recursive resolver, provided by your network or configured in your system or application. It asks for the completed answer. The resolver can use a cached result or follow delegations to find one.

For a lookup without useful cached information, the process is roughly:

Your machine -> Recursive resolver
                    |
                    +-> Root: who handles .com?
                    +-> .com: who handles example.com?
                    +-> Authoritative server: what is its A record?
                    |
Your machine <- Address answer

The root does not keep the address of every website. It directs the resolver toward the name servers for a top-level domain, such as .com. Those servers direct it toward the authoritative servers for example.com. An authoritative server holds the records for the part of the namespace, or zone, it serves.

From your machine’s perspective, the request is recursive: “find the answer for me.” The resolver’s exchanges with those other servers are typically iterative: “give me an answer or a referral.” This division of work is part of DNS’s resolution model ↗.

You can inspect the delegations yourself:

dig +trace example.com A

Here dig follows the delegations; it is not showing a recording of your usual resolver’s work. To inspect just the root’s response:

dig @a.root-servers.net example.com A +norecurse

Look for NS records referring you to .com servers, often with address records that help reach them. The exact server names can vary. Some networks restrict direct DNS queries to outside servers, so +trace can fail even while an ordinary lookup through your configured resolver works.

Caching: Why We Do Not Start at the Root Every Time

A DNS record has a TTL, or time to live, that limits how long it can normally be cached. For example, this is an illustrative answer, not a live address for example.com:

example.com.  300  IN  A  203.0.113.10
               |
          TTL in seconds

A resolver can reuse that record for up to 300 seconds before it normally needs refreshing. Try repeating dig example.com A: a cache’s remaining TTL may count down, although shared resolver infrastructure and refreshes can make the numbers less predictable.

Caching explains why a DNS change does not instantly appear to everyone. It also means different lookups can reuse different parts of the delegation chain.

In the diagrams below, 203.0.113.10 is a documentation address standing in for a server. Use the domain names or loopback addresses in the runnable commands.

IP Addresses: Where Packets Can Go

An IP address gives the network an address to route toward. IPv4 addresses contain 32 bits; IPv6 addresses contain 128 bits:

IPv4: 203.0.113.10
IPv6: 2001:db8::10

“An IP identifies a machine” is a useful first approximation. More precisely, addresses are associated with network interfaces and can also represent virtual services or shared infrastructure. A machine may have several addresses.

A packet carries source and destination addresses. Routers use the destination to make forwarding decisions. They do not need the original domain name to perform ordinary IP forwarding.

Ports: Which Endpoint on That Machine?

A machine can run a web server, an SSH server, and many other programs at once. An IP address alone does not tell the operating system which communication endpoint should receive incoming data.

TCP and UDP add ports, 16-bit numbers from 0 to 65535. Familiar conventions include:

These are conventions, not automatic protocol detection. Sending arbitrary bytes to port 80 does not turn them into HTTP. TCP and UDP also have separate port spaces: TCP port 53 and UDP port 53 can belong to different sockets.

Clients have ports too. When a client opens a TCP connection, the OS usually chooses an ephemeral port, a temporary local port:

Laptop                          Web server
192.168.1.23:52001  ---------->  203.0.113.10:443
192.168.1.23:52001  <----------  203.0.113.10:443

The reply goes to the client’s port 52001. A browser visiting HTTPS sites usually connects to port 443; it does not need to listen on port 443. Browser tabs are not assigned one fixed port each, either. The browser manages connections and assigns their data to requests internally.

Sockets: The Program’s Handle

A socket is an operating-system object representing a communication endpoint. On Unix-like systems, a program accesses it through a file descriptor; a language library often wraps that descriptor in an object.

A TCP client creates a socket, connects it to an address and port, then sends and receives bytes. Name resolution may happen in a library before connecting; the low-level connection operation works with an address.

A TCP server’s setup looks like this Python fragment:

import socket

server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("127.0.0.1", 9330))
server.listen()

conn, addr = server.accept()
# Read and write using conn; call accept() again for another client.

There are two distinct sockets here. server is the listening socket, which waits for connections. accept() returns a connected socket, conn, for one particular peer. The listener remains available to accept more connections. The program must arrange to serve those connections, for example with threads or an event loop.

Think of a reception desk handing each caller to a separate conversation. The desk keeps taking calls. I discuss the OS handles in more detail in What Is a Socket?, but this distinction is enough for our examples.

Binding to 127.0.0.1 makes this an IPv4 loopback service, reachable on the same machine. Binding to 0.0.0.0 instead listens on all local IPv4 addresses; access from other machines still depends on routing and firewalls. 0.0.0.0 is a wildcard bind address, not the address a remote client should use.

Try It: Find the Socket Owner

In one terminal, create a small temporary page and serve it:

network_demo_dir=$(mktemp -d)
printf 'Hello World!' > "$network_demo_dir/index.html"
python3 -m http.server 9330 --bind 127.0.0.1 --directory "$network_demo_dir"

Leave that terminal running. In another terminal:

curl --noproxy '*' -v http://127.0.0.1:9330/index.html
lsof -nP -iTCP:9330

The response body should be Hello World!. --noproxy '*' makes curl connect directly for this experiment, even if proxy environment variables are configured. In lsof, look for the Python process, its PID, its socket descriptor under FD, and the LISTEN state.

The request may finish too quickly to catch an ESTABLISHED connection. To keep one open, run nc 127.0.0.1 9330 in a third terminal without typing a request, then run lsof again. You should now see the accepted connection as well as the listener. Close nc with Ctrl-C. Keep the Python server running for the HTTP and TCP experiments below.

How Do My Bytes Get There?

We now have endpoints. We still need a path between them.

The Routing Table Chooses the Next Hop

Your laptop generally does not work out every router between it and a website. It consults its routing table to choose an outgoing interface and, when needed, a next-hop router. Each router along the way repeats that decision.

Laptop -> Home router -> ISP router -> ... -> Destination network -> Server

Inspect the route your own machine would use:

# Linux / WSL
ip route get 1.1.1.1

# macOS
route -n get 1.1.1.1

An illustrative Linux result is:

1.1.1.1 via 192.168.1.1 dev wlan0 src 192.168.1.23

It means: use source address 192.168.1.23, send through wlan0, and hand the packet to 192.168.1.1 first. Your interface names and addresses will differ, especially inside WSL or when using a VPN.

Routes usually describe address ranges called prefixes. For example, 192.168.1.0/24 covers addresses from 192.168.1.0 to 192.168.1.255; /24 means the first 24 bits are fixed. A simple routing table might contain:

For destination-based routing, the most specific matching prefix wins. A /24 match takes precedence over the /0 default route. The default gateway is the fallback, not necessarily the next hop for every nonlocal address.

Some routes come from directly connected networks or local configuration. Routers can also learn reachability through routing protocols. The details of how those tables are built are covered in my introduction to IP and routing. For following a packet, the key is that each device makes its own next-hop decision.

Local Delivery: Reaching That Next Hop

Suppose the destination IP is 203.0.113.10, but the next hop is your router at 192.168.1.1. How does your laptop deliver the packet to that router?

On an Ethernet link, it places the IP packet inside a frame addressed to the router’s MAC address, a link-layer address. Ethernet and Wi-Fi use MAC addresses for local delivery, although their frame formats differ. A MAC address is not necessarily a permanent factory identity; software can assign or randomize it.

IP destination:        203.0.113.10   (remote server)
Next-hop IP:           192.168.1.1    (local router)
Ethernet destination:  router's MAC  (recipient on this link)

For IPv4, ARP maps an on-link IP address to a MAC address: “who has 192.168.1.1?” The router responds, and the laptop caches the result. IPv6 uses Neighbor Discovery for this job.

Inspect the local neighbor information:

# Linux
ip neigh

# macOS: IPv4 ARP cache
arp -a

You should expect local neighbors here, not the MAC address of a server across the Internet. If the destination is directly on the local link, the frame targets that destination instead of a gateway.

At a router, the incoming link-layer wrapper is removed, and the IP packet is forwarded using a wrapper appropriate for the outgoing link. In ordinary routing without address translation, the destination IP remains the server’s address while the local delivery addresses change at each routed hop.

Try It: Observe Some Hops

traceroute example.com

Traceroute sends probes with increasing IP TTL values, or Hop Limits in IPv6. A router decreases the value; when it reaches zero, the router can send an ICMP Time Exceeded response. Those responses reveal intermediate hops.

This TTL limits how far a packet can travel, unlike the DNS TTL that controls caching time. The same abbreviation serves different purposes.

A * means a probe did not get a response in time. It does not prove that router cannot forward traffic. Routers may filter or limit replies, paths can change, and return paths can differ. Traceroute is evidence about these probes, not a guaranteed map of every future web request.

Shared Links and Best Effort

Packets from many conversations share the same links and router queues. This is packet switching. A TCP connection does not reserve a private physical cable for its lifetime.

Sharing brings delays and failures: a busy queue can delay a packet or overflow and drop it. Packets can arrive out of order or be duplicated. IP does not repair all of that for the application. We will look at the reliability choices after we understand what the application is trying to exchange.

How Do Many Conversations Share the Network?

Your browser, editor, SSH session, and video call may all use the same Wi-Fi connection. Multiplexing combines their traffic onto a shared resource. Demultiplexing separates incoming traffic and hands each part to the appropriate receiver.

The labels that make this possible live in protocol headers.

Encapsulation: Each Layer Adds Its Own Information

For a simple HTTP/1.1 exchange over TCP and Ethernet, the nesting looks like:

Ethernet frame
└── IP packet
    └── TCP segment
        └── Some bytes from the HTTP conversation

Each layer treats data from above as payload and adds information of its own. This is encapsulation. Receiving systems remove wrappers and interpret the relevant fields. One application message can span multiple packets, so this nesting does not imply one HTTP request per TCP segment.

A normal IP forwarding decision needs no HTTP parser. Firewalls and other middleboxes may inspect additional fields, so layering describes responsibilities rather than a promise that nobody can look deeper.

One Server Port, Many TCP Connections

A web server may accept thousands of connections on port 443:

Client A  192.168.1.10:52001 -> 203.0.113.10:443
Client B  192.168.1.11:52002 -> 203.0.113.10:443
Client C  192.168.1.12:52003 -> 203.0.113.10:443

These are endpoint examples before any possible address translation. Within TCP, the four-tuple identifies a connection:

source IP + source port + destination IP + destination port

The destination port alone is insufficient. Each accepted socket has a particular peer, even though all those sockets share the server’s local port. The OS uses the connection information to deliver data to the appropriate socket. This is transport-layer demultiplexing.

Across protocols, tools often use a five-tuple, adding the transport protocol. That separates a TCP flow from a UDP flow using the same address and port numbers.

Multiple Requests Inside One Connection

Sharing also happens above TCP. HTTP/2 can carry multiple application streams on one TCP connection. Its frames contain stream identifiers, allowing a client and server to interleave work on several requests:

One TCP connection
├── HTTP/2 stream 1: /index.html
├── HTTP/2 stream 3: /style.css
└── HTTP/2 stream 5: /app.js

The OS delivers a TCP byte stream to the socket. The HTTP/2 implementation parses frames and uses stream IDs to associate data with requests. These are different demultiplexing decisions. The HTTP/2 specification describes this stream model ↗.

So the number of tabs, requests, sockets, and TCP connections need not be equal. Different layers can share resources independently.

How Does the Receiver Understand My Bytes?

Getting bytes to the right socket is only part of communication. Consider:

47 45 54 20 2f 20 48 54 54 50 2f 31 2e 31

Those hexadecimal values encode GET / HTTP/1.1 in ASCII. The receiver needs an agreement to interpret them as a request rather than an arbitrary string.

A protocol supplies that agreement: the message format, meaning of fields, permitted exchanges, and behavior when something is invalid.

HTTP: Method, Resource, Metadata, Body

HTTP/1.1 makes the agreement easy to inspect because its request line and headers are text:

GET /index.html HTTP/1.1
Host: example.com
User-Agent: curl
Accept: */*
Connection: close

The first line gives the method, request target, and HTTP version. GET asks for a representation of the resource at /index.html. The Host header identifies the intended host, which matters when several websites share an address. Other headers carry metadata.

On the wire, these lines end in \r\n, a carriage return followed by a line feed. An empty line ends the header section. This request has no body; other requests can carry one, such as data submitted to an API. HTTP/1.1 requires Host, and a conforming server rejects a request that omits it. These syntax rules are defined in HTTP/1.1 ↗.

Where Does the Message End?

Here is a simple response, with exactly 12 body bytes and no trailing newline in the body:

HTTP/1.1 200 OK
Content-Type: text/plain
Content-Length: 12

Hello World!

The blank line ends the headers; Content-Length gives the body length in bytes. The receiver can determine that this body is complete after reading Hello World!.

Other HTTP responses use different framing rules, such as chunked transfer coding, and some have no body. A connection close can delimit certain responses. The general lesson is that the protocol defines message boundaries; neither a packet boundary nor one call to recv() does so. HTTP/1.1 specifies the body-length rules explicitly ↗.

Try It: Speak HTTP Without an HTTP Client

With the Python server from the socket experiment still running, send a request using netcat:

printf 'GET /index.html HTTP/1.1\r\nHost: localhost:9330\r\nConnection: close\r\n\r\n' | nc 127.0.0.1 9330

You should see a status line, headers including Content-Length: 12, a blank line, and Hello World!. Python’s basic server normally replies using HTTP/1.0 by default; that is expected here. Connection: close requests closure after the response, so this small experiment has a clear ending.

Change /index.html to /missing and repeat. The connection still succeeds, but the response should be 404 Not Found: transport delivery worked, and the application reported that the requested resource does not exist.

You can also send this style of request to example.com on port 80, changing Host to example.com. Public servers may return redirects or other content. Omitting Host tests a server’s HTTP/1.1 validation; a permissive teaching server may accept it despite the protocol requirement.

How a Stateless Protocol Remembers You

HTTP does not inherently identify a sequence of requests as belonging to one logged-in user. An application can establish that relationship using a cookie or token:

Cookie: session_id=abc123

A server might use that value to find session state stored elsewhere. Another application might expect an Authorization: Bearer ... header and validate the supplied token.

This introduces another kind of name: application-level user or session identity. It is separate from an IP address and from the lifetime of a TCP connection. Opening a new connection does not necessarily log you out, and sharing an IP address does not make two people the same user.

Can I Trust the Peer?

We can now reach a service and exchange meaningful requests. But an answer arriving does not prove that it came from the intended service.

Plain HTTP provides no cryptographic protection against an intermediary reading or modifying its contents. HTTPS protects HTTP using TLS, Transport Layer Security. For the HTTP/1.1 and HTTP/2 connections discussed here, the arrangement is:

HTTP -> TLS -> TCP -> IP -> Local link

TLS provides three distinct properties:

Encryption alone is insufficient. An encrypted connection to an impersonator still delivers your data to the wrong party. TLS combines protection of the data with authentication of the peer; web browsing normally authenticates the server, while client-certificate authentication is optional. See the TLS security model ↗.

Certificates Connect a Name to a Key

A certificate associates a public key with an identity, such as a domain name. For a normal HTTPS connection, the client checks that:

• The certificate covers the hostname it intended to visit.
• The certificate is within its validity period.
• The certificate chain leads to a root certificate trusted by that client.
• The handshake proves possession of the corresponding private key.

The trusted roots come from the client’s trust configuration, which may be provided by the OS, browser, or application. A certificate authority, or CA, participates in issuing the certificates that form that chain.

DNS helps find where to connect. Certificate verification authenticates the intended hostname during the secure connection. A DNS answer pointing somewhere else does not, by itself, give that destination a certificate the client will accept for the original hostname.

This authentication does not establish that a website’s content or business is honest. It establishes the peer’s identity under the client’s trust model.

Try It: Compare HTTP and HTTPS

curl --noproxy '*' -v --max-time 15 http://example.com/
curl --noproxy '*' -v --max-time 15 https://example.com/

In the verbose output, look for address resolution and connection establishment in both cases. In the HTTPS case, also look for TLS negotiation and certificate verification information. Exact messages depend on curl’s TLS backend and version. Curl may negotiate HTTP/2; add --http1.1 if you want to keep the application protocol consistent with the text examples.

Curl displays the request and response because it is an endpoint and can access the plaintext. Seeing that text in curl’s output does not mean an observer on the network can read it.

When an Intermediary Can Inspect HTTPS

An ordinary router or relay can forward encrypted traffic without decrypting HTTP. It can still observe metadata such as addresses, timing, and sizes, and it can disrupt delivery.

An HTTPS inspection proxy instead terminates TLS on both sides:

Client <== TLS connection 1 ==> Inspection proxy <== TLS connection 2 ==> Server

The proxy decrypts the client’s data, can inspect or change it, then encrypts data for the server on the other connection. A debugging tool commonly makes this possible by having its CA added to the client’s trust store, allowing it to issue certificates the client accepts. Compromised credentials or broken verification can also undermine authentication; installing a CA is one mechanism, not the only possible failure.

The question to ask is where the authenticated TLS connection ends. A device being on the path does not automatically make it a TLS endpoint.

What Changes When Traffic Goes Through Someone Else?

Many connections use indirection: an intermediary helps reach or serve the destination. This can provide access to another network, distribute load, apply policy, or serve cached content.

A forwarding proxy makes the extra participant explicit:

Client <--> Proxy <--> Target server

SOCKS5: “Connect for Me”

A SOCKS5 proxy supports a protocol through which a client requests network operations. For its TCP CONNECT operation, the exchange is:

1. The client opens a TCP connection to the proxy.
2. Client and proxy negotiate an authentication method, performing authentication if required.
3. The client supplies a target address and port.
4. The proxy connects to the target and reports success or failure.
5. After success, it relays data in both directions.

The target can be given as an IPv4 address, an IPv6 address, or a domain name. With a domain name, resolution happens on the proxy side. SOCKS5 also defines other operations, including UDP association; the example here concerns TCP relaying. These operations are specified in SOCKS version 5 ↗.

There are now two TCP connections:

Client:52001  <-->  Proxy:1080
Proxy:53002   <-->  Target:443

The server sees the proxy’s outgoing connection as its TCP peer, possibly after further address translation. The proxy keeps an accepted client socket paired with an outgoing target socket. With many clients, keeping those pairs correct is an application-level responsibility on top of the OS’s socket demultiplexing.

A SOCKS5 Relay and an HTTPS Inspection Proxy

For HTTPS through an ordinary SOCKS5 TCP relay, the TCP and TLS boundaries are different:

TCP:   Client <----> SOCKS5 proxy <----> Server
TLS:   Client <=======================> Server

The client performs TLS with the target through the relay and verifies the target’s certificate. The relay copies encrypted bytes. Splitting TCP into two connections does not itself split TLS into two sessions.

With an inspection proxy, the TLS picture is instead:

TLS:   Client <====> Inspection proxy <====> Server

This distinction explains why adding a SOCKS5 proxy does not automatically let it read an HTTPS request. It can see the requested destination and relay metadata, but cannot simply decrypt the HTTP content. SOCKS5 also does not itself give ordinary relayed traffic encryption: plain HTTP remains plain unless another protection is added.

Try It: Choose Who Resolves the Name

If you already have a SOCKS5 proxy listening at 127.0.0.1:1080, compare:

# Direct connection
curl --noproxy '*' -v --max-time 15 https://example.com/

# Through SOCKS5; curl resolves the target name locally
curl --noproxy '' --socks5 127.0.0.1:1080 -v --max-time 15 https://example.com/

# Through SOCKS5; the proxy resolves the target name
curl --noproxy '' --socks5-hostname 127.0.0.1:1080 -v --max-time 15 https://example.com/

These commands use an existing proxy; they do not start one. Without a listener at that address, the proxy connection will fail.

Look for the initial connection to port 1080, the SOCKS negotiation, and then TLS with the target. --noproxy '' clears proxy bypass rules for these runs. Curl documents the resolution difference between --socks5 and --socks5-hostname ↗.

Changing where resolution happens can change reachability and which resolver observes the query. The intended HTTPS identity remains example.com in both cases.

Other Forms of Indirection

These systems change different parts of the path:

These are not all equivalent to a SOCKS5 relay. Ordinary NAT translates packets without becoming a TCP endpoint. A VPN can tunnel IP packets without terminating the TCP connections inside them, and a split-tunnel VPN carries only selected routes. A CDN may intentionally terminate TLS as an authorized endpoint for the website.

To understand a particular setup, draw the actual connections and mark where TCP, TLS, and the application protocol end. That is more informative than calling every box in the middle “a proxy.”

What If the Network Loses, Reorders, or Splits My Data?

We have been describing the conversation as if bytes simply arrived. The network underneath it is still best effort. Applications need to choose which guarantees they want and which failures they will handle themselves.

TCP: A Reliable, Ordered Byte Stream

TCP supplies a reliable, ordered byte stream between endpoints. Several mechanisms work together:

Flow control and congestion control address different bottlenecks: the receiving endpoint and the network path. TCP handles retransmission and ordering, but a connection can still fail. It does not promise that every attempted transfer will eventually succeed. These responsibilities are described in the TCP specification ↗.

Your Writes Are Not Message Boundaries

Suppose a sender writes these bytes successfully:

sock.sendall(b"hello")
sock.sendall(b"world")

The receiver could observe:

recv() -> b"helloworld"

or:

recv() -> b"hel"
recv() -> b"lowor"
recv() -> b"ld"

In both cases, concatenating the received bytes produces helloworld. TCP preserves order, not the boundaries of the sender’s calls. A receive buffer size is a maximum for that call, not a requested message length.

This is why the HTTP body-length rules mattered earlier. The receiver needs a parser that accumulates bytes and recognizes complete messages. Common approaches include fixed-size messages, delimiters, and length prefixes:

[ length = 5 ][ hello ][ length = 5 ][ world ]

Even a length field can arrive across multiple reads. Your program keeps incomplete data in a buffer until enough arrives to parse the next part. A SOCKS5 greeting, a TLS record, and an HTTP response all have their own framing rules above the TCP stream.

Try It: Read Only Ten Bytes at a Time

With the local Python HTTP server still running, save this as read_stream.py and run python3 read_stream.py:

import socket

request = (
    b"GET /index.html HTTP/1.1\r\n"
    b"Host: localhost:9330\r\n"
    b"Connection: close\r\n"
    b"\r\n"
)

with socket.create_connection(("127.0.0.1", 9330), timeout=5) as sock:
    sock.sendall(request)
    chunks = []
    while True:
        data = sock.recv(10)
        if not data:
            break
        print(repr(data))
        chunks.append(data)

print("\nReassembled response:")
print(b"".join(chunks).decode("utf-8"))

Each printed chunk contains at most ten bytes. A status line, header, or body can cross several reads. Joining the chunks reconstructs the response. The chunk boundaries are application reads, not observations of the network’s packet boundaries.

sendall() handles partial writes for this small blocking example. recv(10) can return fewer than ten bytes; a return of b"" indicates that the peer has closed its sending direction after the buffered data has been consumed. The timeout keeps the example from waiting indefinitely. These behaviors are documented in Python’s socket API ↗.

We read until closure here because the request explicitly asks the server to close. A general HTTP client must parse HTTP framing instead of assuming every response ends with the connection. You can now stop the local server with Ctrl-C.

Packets, Segments, and Messages Have Different Boundaries

A link has an MTU, a maximum size for the packet it can carry. An ordinary Ethernet IP MTU is often 1500 bytes, including the IP header. TCP divides its byte stream into segments sized for the path; their payloads must leave room for the relevant headers.

This segmentation is distinct from IP fragmentation, which splits an IP packet. IPv4 can permit routers to fragment packets; IPv6 routers do not fragment them. Neither mechanism defines the messages your application sees.

Keep three units separate: IP moves packets, TCP exposes a byte stream, and an application protocol defines messages. One message can span many packets, and one read can include bytes from multiple messages.

UDP: Datagram Boundaries with Fewer Guarantees

UDP delivers datagrams. Separate datagrams retain separate boundaries when received, but UDP has no built-in connection handshake, retransmission, or ordering guarantee. An application can call connect() on a UDP socket to select a peer; that does not create a TCP-style handshake or reliable connection.

Your receive buffer must be large enough for the datagram: with typical socket APIs, an undersized buffer can truncate the data. Applications must also avoid assuming arbitrarily large datagrams will travel successfully across a path.

The right choice depends on the application:

UDP does not require an application to abandon reliability. QUIC builds secure, reliable streams over UDP ↗, and HTTP/3 uses that transport. This is why “web means TCP” and “UDP means unreliable application” are both incomplete shortcuts.

What Your Program Still Has to Decide

Even with TCP, a program needs to handle timeouts, partial reads, failed writes, closure, and resets. A successful socket write means the local networking stack accepted the bytes; it does not prove the remote application processed the request. Even a TCP acknowledgment is not an application-level success response.

Suppose you submit an operation and the connection fails before its response arrives. The server may have completed it, or may never have received it. A retry policy must consider whether repeating the operation is safe, perhaps using an application-defined request identifier to avoid duplicate work.

Proxies also need backpressure. If a proxy reads from a fast client while its target is slow, buffering everything indefinitely will exhaust memory. It must bound buffers and pause reads when the outgoing side cannot keep up. Relaying bytes involves managing rates and lifetimes as well as pairing sockets.

Follow One Request from Start to Finish

Return to the command we began with. To make the path explicit, request HTTP/1.1 directly:

curl --noproxy '*' --http1.1 -v --max-time 15 https://example.com/index.html

For a fresh connection, with no reusable connection already available:

1. Curl interprets the URL: HTTPS, hostname example.com, default port 443, path /index.html.
2. Name resolution supplies candidate addresses, possibly from a cache. Curl selects an address to try.
3. The OS chooses a source address, a local port, an outgoing interface, and a route. On a local Ethernet or Wi-Fi link, neighbor resolution supplies the necessary next-hop link address if it is not cached.
4. TCP establishes a connection. Its packets travel through local delivery and successive routing decisions; the remote OS associates the connection with the server’s listening endpoint.
5. Curl negotiates TLS with the peer and verifies its certificate against the intended hostname and trust configuration.
6. Curl sends the HTTP request inside TLS. The server reads the request and uses its host, path, method, and other fields to decide how to respond.
7. The response returns through the network. TCP presents bytes in order, TLS verifies and decrypts them, and HTTP parsing separates headers from the body and determines completion.
8. Curl displays the result. Connections can close or be reused according to the protocols and client behavior.

Caching, connection reuse, proxies, and HTTP/3 change parts of this sequence. We can reason about them by identifying which responsibility changed.

That also gives us a way to investigate a failure:

When I get lost in networking, I go back to the program that is waiting: what does it know, what is it trying to do, and which layer is supposed to provide the next piece? Looking at one lookup, one socket, or one request with that question in mind makes the terminology much easier to use.