Sitemap
Open in app

Sign in

Medium Logo
Write
Search

Sign in

eBPF in Go: Observability for AI-Generated Services

From blind debugging to kernel-level visibility in 10 minutes

8 min readJun 25, 2026

--

I shipped a service in March that looked fine in staging. P95 latency was 40ms, errors were under 0.1%, everything green. Two weeks later, on a Tuesday afternoon during our weekly demo, the service started timing out. Not gradually. Suddenly. P95 jumped to 4 seconds in under a minute.

I had no idea why.

The Go service was clean. Standard net/http, a goroutine pool, Redis for caching. I'd even added Prometheus metrics. But the metrics showed nothing useful. CPU was normal. Memory was normal. GC pauses were normal.

The problem was in the kernel. Specifically, in how my service interacted with the network stack, the file system, and other kernel subsystems. Prometheus couldn’t see any of it because those interactions happen below the Go runtime.

That’s when I started learning eBPF. And that’s why I’m writing this.

If you use AI to generate Go services, you have a visibility problem. The Go runtime can tell you what your code is doing. eBPF can tell you what your code is doing to the kernel. You need both.

This tutorial shows you how to add eBPF-based observability to a Go service using Cilium’s eBPF library. I’ll explain what eBPF is, why it matters for AI-generated code, and walk through a working example you can run in under 10 minutes.

Press enter or click to view image in full size

What is eBPF?

eBPF (extended Berkeley Packet Filter) is a technology that lets you run sandboxed programs inside the Linux kernel without modifying kernel source code or loading kernel modules. The kernel was first extended to support eBPF in Linux 3.18 (2014), and the technology has matured significantly since then.

According to the eBPF Foundation, eBPF programs can be attached to various kernel hooks: network events, system calls, function entry/exit points, and more. They execute safely in the kernel and can pass data to userspace programs through shared maps.

In practice, this means you can observe and modify kernel behavior from userspace, without recompiling the kernel or loading risky kernel modules.

For Go services, eBPF lets you see:

  • Which syscalls your service makes and how long they take
  • TCP connections, retransmissions, and drops
  • File system operations
  • Security events (privilege escalation attempts, unusual syscalls)

Why AI-Generated Code Makes eBPF More Important

When I write Go code by hand, I have context. I know which libraries I chose and why. I know which syscalls they’ll trigger. I can reason about the kernel interactions.

When an LLM writes Go code, I don’t have that context.

This is where eBPF comes in. It gives you X-ray vision into what your AI-generated service is actually doing at the kernel level. You can see the syscall patterns, the network traffic, the I/O behavior, even if you didn’t write the code yourself.

Prerequisites

You need:

  • Linux kernel 4.19+ (for stable eBPF support)
  • Go 1.21+
  • clang and llvm (for compiling eBPF programs)
  • bpftool (for loading eBPF programs)

On Ubuntu:

sudo apt install clang llvm libbpf-dev linux-headers-$(uname -r) bpftool

Verify your kernel supports eBPF:

uname -r  # Should be 4.19 or higher

Step 1: Install the eBPF Go Library

Cilium’s eBPF is the most mature Go library for eBPF. It’s used in production by Cilium, Falco, and many other projects.

go get github.com/cilium/ebpf
go get github.com/cilium/ebpf/link
go get github.com/cilium/ebpf/perf

Step 2: Write an eBPF Program

eBPF programs are written in C and compiled to bytecode. Create a file called trace.c:

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>

struct event_t {
    u32 pid;
    u64 timestamp;
    char comm[16];
};

struct {
    __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
    __uint(key_size, sizeof(u32));
    __uint(value_size, sizeof(u32));
} events SEC(".maps");

SEC("tracepoint/syscalls/sys_enter_openat")
int trace_openat(struct trace_event_raw_sys_openat *ctx) {
    struct event_t event = {};
    u64 pid_tgid = bpf_get_current_pid_tgid();
    event.pid = pid_tgid >> 32;
    event.timestamp = bpf_ktime_get_ns();
    bpf_get_current_comm(&event.comm, sizeof(event.comm));

    bpf_perf_event_output(ctx, &events, BPF_F_CURRENT_CPU,
                          &event, sizeof(event));
    return 0;
}

char _license[] SEC("license") = "GPL";

This program traces the openat syscall (used for file opens) and sends events to userspace.

You can always ask your LLM to update the C program to track the system actions you want.

(See end of post for link to Github with full source code)

Step 3: Load and Attach the eBPF Program

Before Go can load the eBPF program, it needs generated bindings for the C code. The easiest way to do that is with bpf2go.

Because go generate only scans .go files, place the following directive at the top of your main.go:

//go:generate go run github.com/cilium/ebpf/cmd/bpf2go -cc clang -target amd64 trace ./bpf/trace.c -- -I./bpf

This isn’t Go code. It’s an instruction for the go generate command. When you run:

go generate ./...

Go searches for //go:generate directives and executes the commands it finds. In this case, it invokes bpf2go, which compiles trace.c into eBPF bytecode using clang and generates Go bindings that make the resulting program accessible from your application.

The generated files (trace_x86_bpfel.go and trace_x86_bpfel.o) contain helper types and loader functions, allowing your Go code to treat the eBPF program almost like any other package dependency. Without this directive, you'd need to rerun bpf2go manually every time you modified the eBPF source.

With code generation in place, the Go application can load the compiled eBPF objects, attach them to a kernel tracepoint, and begin receiving events:

//go:generate go run github.com/cilium/ebpf/cmd/bpf2go -cc clang -target amd64 trace ./bpf/trace.c -- -I./bpf
package main

import (
   "bytes"
   "encoding/binary"
   "fmt"
   "log"
   "os"
   "os/signal"
   "syscall"
  
   "github.com/cilium/ebpf/link"
   "github.com/cilium/ebpf/perf"
   "github.com/cilium/ebpf/rlimit"
)

func main() {
 // Allow the process to lock memory for eBPF maps
 if err := rlimit.RemoveMemlock(); err != nil {
    log.Fatal(err)
 }

 // Load pre-compiled eBPF program
 objs := traceObjects{}
 if err := loadTraceObjects(&objs, nil); err != nil {
    log.Fatal(err)
 }
 defer objs.Close()

 // Attach to the openat tracepoint
 tp, err := link.Tracepoint("syscalls", "sys_enter_openat",
  objs.TraceOpenat, nil)
 if err != nil {
    log.Fatal(err)
 }
 defer tp.Close()

 // Open perf reader
 reader, err := perf.NewReader(objs.Events, 4096)
 if err != nil {
    log.Fatal(err)
 }
 defer reader.Close()

 fmt.Println("Observing file opens... Press Ctrl+C to exit.")

 sig := make(chan os.Signal, 1)
 signal.Notify(sig, syscall.SIGINT, syscall.SIGTERM)

 go func() {
    for {
       record, err := reader.Read()
       if err != nil {
          continue
       }
    
       var event struct {
          PID       uint32
          Timestamp uint64
          Comm      [16]byte
       }
       if err := binary.Read(bytes.NewReader(record.RawSample),
          binary.LittleEndian, &event); err != nil {
          continue
       }
    
       fmt.Printf("PID %d (%s) opened a file\n",
        event.PID, string(event.Comm[:]))
    }
 }()

 <-sig
}

The startup sequence is straightforward:

  1. Remove the default memory-lock limit so eBPF maps can be created.
  2. Load the generated eBPF objects using loadTraceObjects().
  3. Attach the TraceOpenat program to the sys_enter_openat tracepoint.
  4. Open a perf event reader connected to the eBPF event map.
  5. Continuously read events, decode them into a Go struct, and print the process information.

At this point, every time a process calls openat(), the kernel executes the attached eBPF program, which emits an event into the perf buffer. The Go application reads those events in user space and turns them into human-readable output.

Step 4: Compile the eBPF Program

Generate the Go bindings:

go generate ./...

This produces trace_x86_bpfel.go and trace_x86_bpfel.o.

Step 5: Run and Observe

go build -o ebpf-workshop ./
sudo ./ebpf-workshop # sudo required for access to memlock

Open another terminal and trigger some file opens:

cat /etc/hostname
ls /tmp

You’ll see output like:

PID 252006 (�"cat) opened a file
PID 252006 (�"cat) opened a file
PID 252006 (�"cat) opened a file
PID 252006 (�"cat) opened a file
PID 252016 (�"ls) opened a file
PID 252016 (�"ls) opened a file
PID 252016 (�"ls) opened a file

Now run your Go service. You’ll see every file it opens, with the PID and process name.

You can also paste the output of the program and ask your LLM to find any anomalies.

What You Can Observe

With eBPF, you can trace:

Press enter or click to view image in full size

For a comprehensive list, see the bpftrace reference.

Real-World Use Cases

Use Case 1: Finding Unexpected Syscalls

I ran my service with eBPF tracing and discovered it was making 200+ openat calls per second to read config files that should have been cached. The AI-generated code had chosen to re-read configs on every request instead of caching them. I would never have caught this from application-level metrics alone.

Use Case 2: Network Latency Diagnosis

When my service started timing out in March, eBPF showed me that the kernel was dropping TCP packets due to a misconfigured net.core.rmem_max setting. The Go runtime had no visibility into this.

Use Case 3: Security Monitoring

You can use eBPF to detect privilege escalation attempts, unusual syscalls, or connections to suspicious IPs. Falco uses this approach for runtime security.

And with the onslaught of Chinese cyber attacks, now is the best time to observe what our programs are doing at the kernel level.

Limitations

eBPF has trade-offs:

  1. Linux and Windows-only. It doesn’t work on macOS. For cross-platform observability, you’ll need a fallback.
  2. Requires root or CAP_BPF. You need elevated privileges to load eBPF programs.
  3. Kernel version sensitivity. Some features require newer kernels. The Cilium kernel requirements page lists which features need which versions.
  4. Verifier overhead. The eBPF verifier checks programs for safety, which adds startup latency.

Project Structure

Here’s how I structure observability for my services:

myservice/
├── main.go
├── internal/
│   └── observability/
│       ├── trace.c
│       ├── trace_bpfeb.go
│       ├── trace_bpfel.go
│       └── observer.go
└── go.mod

The observer.go file wraps the eBPF loading and exposes a simple API:

type Observer struct {
    reader *perf.Reader
}

func NewObserver() (*Observer, error) { /* ... */ }
func (o *Observer) Events() <-chan Event { /* ... */ }
func (o *Observer) Close() error { /* ... */ }

Next Steps

To go deeper:

  1. Read the Cilium eBPF documentation — comprehensive and well-maintained.
  2. Try bpftrace — a high-level tracing language for eBPF. Great for ad-hoc investigation.
  3. Explore Pixie — a Kubernetes observability platform built on eBPF. Useful for production deployments.
  4. Check out eBPF Summit talks — the conference has excellent presentations on real-world use cases.

Conclusion

If you use AI to generate Go services, you need “observability” at the kernel level. Application-level metrics tell you what your code is doing. eBPF tells you what your code is doing to the kernel. You need both to debug performance issues, diagnose latency problems, and catch security-relevant patterns.

The Cilium eBPF library makes it straightforward to add kernel-level observability to any Go service. The setup takes about 10 minutes, and the visibility you gain is substantial.

I now ship every production service with eBPF tracing enabled. It’s caught multiple issues that I never would have seen with application-level metrics alone.

Give it a try. Once you see kernel-level data, you can’t go back.

Relevant Links

Source Code

Core eBPF Resources

Tools Mentioned

Additional Reading

Golang
Ebpf
AI
Observability
System Programming

--

--

Cheikh seck
Cheikh seck

Written by Cheikh seck

Get your personal AI intern: https://liteagent.cloud/

Help

Status

About

Careers

Press

Blog

Store

Privacy

Rules

Terms

Text to speech