ztp/docs/06-telemetry.md

+++ title = "Telemetry: Logging and Analytics" date = 2023-03-20T17:38:12Z weight = 6 +++

Chapter 4: Telemetry

In Chapter 4, Palmieri focuses on logging and telemetry. Axum is very different from Actix, and my first foray into trying to understand it led me to Tower, the Rust community's de-facto standards for modular networking development and design.

I completely short-circuited much of what the book recommended and, instead, just went with the most basic implementation possible. I added the tracing libraries as recommended by the Axum developers, and then implemented the first level of tracing as recommended by Tower:

$ cargo add --features tower_http/trace,tracing tower tower_http tracing tracing_subscriber

And then I updated the app startup code to include it:

pub async fn app(configuration: &Settings) -> Router {
    tracing_subscriber::fmt::init();

    let pool = PgPoolOptions::new()
        .max_connections(50)
        .connect(&configuration.database.url())
        .await
        .expect("could not connect to database_url");

    routes().layer(Extension(pool)).layer(TraceLayer::new_for_http())
}

That is literally all that was needed. And the output is:

2023-03-25T16:49:06.385563Z DEBUG request{method=GET uri=/ version=HTTP/1.1}:
    tower_http::trace::on_request: started processing request
2023-03-25T16:49:06.386270Z DEBUG request{method=GET uri=/ version=HTTP/1.1}:
    tower_http::trace::on_response: finished processing request latency=0 ms
    status=200

That's not great logging, but it's a start. As I understand it, tracing_subscriber::fmt::init() initializes the formatter, but I'm confused as to where this is saved or stored, since it seems to be... nowhere. The deeper Rust gets, the wilder it seems.

What I did manage was to create, as recommended by Chris Allen, a very simple Layer that shoves a new object into the collection of data being passed around by the request. That object contains a unique UUID for the session being processed. Since Tokio is a multi-threaded system, having a UUID allows us to trace each individual request from beginning to end... provided I've hooked up by handlers just right.

I learned most of this by reading the Axum Session source code, which implements something much more complex. Since we're at a deeper level of the service handling I need a function takes a Request and returns a Response, and in the middle inserts a SessionId into the Request passed in; by giving the type a name any handlers can now find and use that SessionId:

/// In file `session_id.rs`
#[derive(Clone)]
pub struct SessionId(pub Uuid);

pub async fn session_id<B>(mut req: Request<B>, next: Next<B>)
  -> Result<Response, StatusCode> {
    req.extensions_mut().insert(SessionId(Uuid::new_v4()));
    Ok(next.run(req).await)
}

With that, I now need to add it to the layers initialized with the app object:

/// In lib.rs:pub async fn app()`:
    routes()
        .layer(Extension(pool))
        .layer(TraceLayer::new_for_http())
        .layer(middleware::from_fn(session_id::session_id))

And with that, the SessionId is available. Since it's the outermost layer, it can now be used by anything deeper in. Let's add it to the subscribe function:

/// In routes/subscribe.rs/subscribe()
pub(crate) async fn subscribe(
    Extension(session): Extension<SessionId>,
    Extension(pool): Extension<PgPool>,
    payload: Option<Form<NewSubscription>>,
) -> Result<(StatusCode, ()), ZTPError> {
    if let Some(payload) = payload {
        // Multi-line strings in Rust. Ugly. Would have preferred a macro.
        let sql = r#"INSERT INTO subscriptions
            (id, email, name, subscribed_at)
            VALUES ($1, $2, $3, $4);"#.to_string();
        let subscription: Subscription = (&(payload.0)).into();

        tracing::info!(
            "request_id {} - Adding '{}' as a new subscriber.",
            session.0.to_string(),
            subscription.name
        );
     // ...

And with that, every Request now has a strong ID associated with it:

2023-03-26T22:19:23.305421Z  INFO ztp::routes::subscribe:
    request_id d0f4a6e7-de0d-48bc-902b-713901c1d63b -
    Adding 'Elf M. Sternberg' as a new subscriber.

That's a very noisy trace; I'd like to start knocking it down to something more like a responsible log, or give me permission to format it the way I like. I'm also getting incredibly noisy messages from the sqlx::query call, including the text of the SQL template (the let sql = ... line above), which I really don't need every time someone makes a request, and is horribly formatted for principled analytics.

Configuring it to return JSON turned out to be easy, although my first pass puzzled me. I had to turn json formatting on as a feature:

$ cargo add --features=json tracing_subscriber

And then it was possible to configure the format:

    // in lib.rs:app()
    // ...
    let format = tracing_subscriber::fmt::format()
        .with_level(false) // don't include levels in formatted output
        .with_thread_names(true)
        .json(); // include the name of the current thread

    tracing_subscriber::fmt().event_format(format).init();
    // ...

{
  "timestamp":"2023-03-26T22:53:13.091366Z",
  "fields": {
    "message":"request_id 479014e2-5f13-4e12-8401-34d8f8bf1a18 - "
              "Adding 'Elf M. Sternberg' as a new subscriber."},
 "target":"ztp::routes::subscribe",
 "threadName":"tokio-runtime-worker"
}

Conclusion

This pretty much concludes my week-long foray into Palmieri's book; I'm not going to worry too much about the deployment stuff, since that's part of my daytime job and I'm not interesting in going over it again.

Overall, this was an excellent book for teaching me many of the basics, and provides a really good introduction into the way application servers can be written in Rust. I disagree with the premise that "the language doesn't mean anything to the outcome," as I've heard some people say, nor do I think using Rust is some kind of badge of honor. Instead, I think it's a mark of a responsible developer, one who can produce code that works well the first time, and with some hard thinking about how types work (and some heavy-duty exposure to Haskell), Rust development can be your first thought, not your "I need speed!" thought, when developing HTTP-based application servers.