|
Get a state-of-the-art, fully assembled agent harness. (Sponsored)
 |
Strands harness is the open-source, state-of-the-art, fully assembled agent harness. It’s a complete, production-ready agent the moment you install it. Across six benchmarks it matched or beat other harnesses on accuracy while using 28% fewer tokens.
You get shell, file, and web tools, plus context management, memory, and subagents, all tuned and ready to use. Every default is yours to change. Run it as a library or from the strands CLI, on any model and any cloud. It’s built on the open-source Harness SDK, so you can extend or replace anything.
This week’s system design refresher:
How SSH Works
Top 6 Techniques to Make Your AI System Efficient
The Anatomy of an AI Agent
How do you know if your AI app actually works?
How SSH Works
Secure shell is a method to access a remote machine securely over an unsecured network. It starts with a TCP connection from the SSH client to the remote machine (SSH server).
 |
Both the machines exchange the SSH versions and negotiate crypto algorithms, and each side runs a key exchange protocol. The SSH server sends the public key and a signature, which the client machine verifies. That host key is then checked against the known_hosts file.
Both the machines then derive session keys on their end and do not share them over the network. Same keys, computed separately, never put on the wire. Then the SSH client starts authentication by sending the public key for login.
The SSH server matches the public key in the authorized_keys, and the SSH client also signs the auth request with the private key and sends the digital signature to the SSH server.
The private key never leaves your machine. The server will verify the signature with the client public key. This completes the SSH authentication, and now the session is open for communication.
Top 6 Techniques to Make Your AI System Efficient
Serving LLMs at scale is expensive. These 6 popular techniques reduce latency and cost, and make your system more efficient.
 |
Streaming: The model sends each token as soon as it becomes ready. This changes the user’s perceived latency to time-to-first-token.
Quantization: Convert the model's weights to lower precision like FP8. This conversion leads to less memory usage, which translates to faster and cheaper serving.
Continuous batching: With this technique, new requests are added to the batch once a slot becomes available. Therefore, the GPU is less idle.
Prefix caching: This technique caches internal calculations for given input prompts. At runtime, if the prompt has a cached prefix, those calculations are skipped.
Paged KV cache: This technique stores the cache in fixed-size blocks instead of one contiguous chunk. At runtime, blocks are allocated as the sequence grows, so memory is not reserved up front.
Speculative decoding: Here, a small model (speculator) proposes multiple tokens. After that, the main model verifies them in one forward pass.
Over to you: what is missing from this list?