Skip to main content
Be Bitwise
RustAsync & Networking

Building an HTTP Client

We'll build a small HTTP client from raw TCP, then hold it up against the libraries everyone actually uses.

Raw HTTP over TCP

rust
use std::io::{Read, Write};
use std::net::TcpStream;
fn http_get(host: &str, path: &str) -> Result<String, Box<dyn std::error::Error>> {
let addr = format!("{}:80", host);
let mut stream = TcpStream::connect(&addr)?;
let request = format!(
"GET {} HTTP/1.1\r\nHost: {}\r\nConnection: close\r\n\r\n",
path, host
);
stream.write_all(request.as_bytes())?;
let mut response = String::new();
stream.read_to_string(&mut response)?;
Ok(response)
}

Those \r\n pairs are the whole framing protocol, and they're easy to get wrong when you're staring at a Rust string literal. The builder below lays out just the request line as bytes. Expand the HTTP layer and find 0d 0a at the end, then type a longer path into the Path field (hex, so /index.html is 2f696e6465782e68746d6c) and watch the line grow. Every header after it follows the same pattern, Name: value plus 0d 0a, and one extra 0d 0a on its own tells the server the headers are over.

Network Packet Builder

16 bytes
Hex Dump
474554202f20485454502f312e310d0a

Click a layer to expand fields. Hover a field to highlight its bytes.

This client is deliberately minimal, so know where it stops working. It speaks plain HTTP on port 80 only: an https:// site wants a TLS handshake first, which is why the capstone tool turns https:// URLs away. It finds the end of the response by waiting for the server to close the connection, which is what Connection: close asks for. It hands back the body exactly as it arrived, so a server that replies with Transfer-Encoding: chunked leaves hex chunk sizes mixed into your text. And read_to_string fails with InvalidData the moment the body isn't valid UTF-8, an image say; read into a Vec<u8> with read_to_end if you need arbitrary bytes.

To see the real thing, run nc -l 8080 in one terminal, change the :80 in http_get to :8080, and call http_get("127.0.0.1", "/"). Netcat prints exactly what your code sent. It also never replies, so press Ctrl-C to let your client finish.

Parsing the Response

rust
fn parse_response(raw: &str) -> (&str, &str) {
match raw.find("\r\n\r\n") {
Some(i) => (&raw[..i], &raw[i + 4..]),
None => (raw, ""),
}
}
fn get_status_code(headers: &str) -> Option<u16> {
let first_line = headers.lines().next()?;
let parts: Vec<&str> = first_line.split_whitespace().collect();
parts.get(1)?.parse().ok()
}
Key Concept

Hand-rolling an HTTP client over raw TCP demystifies what reqwest is doing for you. The wire format is plain text: request line, CRLF-separated headers, a blank line, then the body. Parsing it is just string slicing.

URL Parsing

rust
struct Url {
host: String,
port: u16,
path: String,
}
fn parse_url(url: &str) -> Option<Url> {
let without_scheme = url.strip_prefix("http://")?;
let (host_port, path) = match without_scheme.find('/') {
Some(i) => (&without_scheme[..i], &without_scheme[i..]),
None => (without_scheme, "/"),
};
let (host, port) = match host_port.find(':') {
Some(i) => (&host_port[..i], host_port[i + 1..].parse().ok()?),
None => (host_port, 80u16),
};
Some(Url {
host: host.to_string(),
port,
path: path.to_string(),
})
}

The copy below swaps strip_prefix for starts_with and slicing so it builds on the runner's older compiler, then feeds it six URLs. Two of the answers are wrong in ways a real client would regret.

Try itMake parse_url reject an empty host, then change it to return Result<Url, String> so each rejection says why.
Rust (1.40.0)

Error Types

A real HTTP client breaks in many places, so you want a single error type that captures the lot:

rust
#[derive(Debug)]
enum HttpError {
ConnectionFailed(std::io::Error),
InvalidUrl(String),
InvalidResponse(String),
Timeout,
}
impl From<std::io::Error> for HttpError {
fn from(e: std::io::Error) -> Self {
HttpError::ConnectionFailed(e)
}
}

Comparison: Hand-Written vs Library

Our hand-written client:

rust
let response = http_get("example.com", "/")?;

Using the reqwest crate:

rust
let response = reqwest::get("http://example.com/").await?.text().await?;

reqwest covers connection pooling, redirects, cookies, TLS, compression, timeouts, and a long tail of edge cases. You now know what's hiding behind that single function call.

What You Have Learned

This module walked through Rust's async system, TCP networking, concurrency primitives, and HTTP fundamentals. The capstone project leans on all of them to build a working CLI tool.

Challenges

Parse a simple URL string and extract the host and path.

Rust (1.40.0)