Exploring High-Performance Experimentation with Notion, Statig, and Vercel

everybody alrighty welcome folks um as everyone is getting kind of oriented and popping in I want to have everybody

who's uh dialing in uh let's go ahead and hop into the chat and let us know where you're calling from um where you

got where you coming from what's what's hot on your mind today anything interesting going on as folks are piling

in we're going to give it a minute uh we did this a couple weeks ago um for the folks who are with me uh Joe Carolyn and

Scott I would love to know right now if we were to have like an office Olympics or like a bar Olympics what what would

you dominate in Connect for Uno um you know we've had some folks say like uh like Bard die folks in the audience let

us know where you're from let us know what like childhood game that you would come in and dominate like Mario Kart um

Sky Joe Carolyn any any bar games that you would just like totally Crush right now if we were to do this um anything

coming top of mind our office always runs uh ping pong tournaments um ping pong tournament okay

that's my go-to and we also do a paper plane competitions but that's like my big weakness like I cannot my life this

is absolutely different that's real that's like a physics challenge Joe that's not bar Olympics but that is fun

uh when I was a kid we did this egg drop thing where you would go like I don't know like two or three stories up and

like Drop The Egg and like like You' build like a little container for an egg with like a parachute or something and

you could see if it would like live uh through the drop now this is like I think that's just what the stock market

is now is like everyone just going dropping eggs um so um Carol and Sky yeah caroly and Sky tell me what your uh

what your bar bar game is while everybody's kind of piling in if you're just joining us folks we're we're we're

we're kind of fill on the space here while everyone's coming in to talk about bar games and um or like office Olympics

Carolyn and Sky I need an answer from you guys Connect Four uh Four Square I would put as a bar game

um what's what's the answer here yeah for me it' probably be D Jenga especially I think the more the more the

stakes are high the better so sky where where what do you have for us here um I'm going Connect Four I

recently did like a speed Connect 4 Tournament with some friends where you get a like oh wow 30 second chess clock

and you you know have to finish a Connect 4 game at 30 seconds really fun for anyone who's looking for like a you

know low effort uh competitive game that everyone knows how to play and it's pretty good at got to do it got to do it

okay I love it I think mine would be um in I'm gonna get roasted for saying this mine has got to be um bags or corn holes

what it's called um this was invented in my hometown and I am I think I think was just like everyone knows how to be very

good at it from this like it's just like born into the water like everyone just knows how good like you just become good

at it automatically Without Really playing that much I think I would I think I would have to do that if we had

a bar Olympics or like office Olympics but we're not here to talk about that this is just kind of filling some space

while everybody's piling in uh if you're new to us uh let us know where you are let us know where you're calling from

any questions you have along the way let's do some housekeeping stuff any questions you have along the way drop

that into our Q&A box over here on the side um and what we'll do is we'll be able to go ahead and just like grab this

uh grab these questions over time and then drop those in and we'll be able to answer those later um so stick with us

here for about 45 minutes uh we're going to go over quite a few things um and we're going to be talking about how

notion achieves high performance experimentation with stat IG and versel um my name is Matt Jared I'm located in

Austin Texas um and I'm with versel so I'm GNA kind of give you guys kind of like a an overview of where we are today

here um when we get into our agenda uh but before we do that I want to talk about the folks who we're working with

today I've got Carolyn uh from notion Joe and sky from static and I'll give a little bit introduction on myself here

in a second but Carolyn you're kind of our our guest of honor here let us know where are you in your career how long

have you been at notion what's the story uh you leave a pretty you lead a pretty impressive software engineering kind of

like role at notion in the marketing engineering side which we're going to hear all about but I kind of want to

hear your hero's journey like uh what how what kind of roles have you done so far in your career to get to where we

are today so yeah originally from Houston Texas currently in San Francisco uh been

working as an engineer for uh probably about six and a half years and yeah I just celebrated my one year at notion

very excited about that nice way to go love that thank you and like you said work on the marketing engineering team

here um uh my background's primarily been in back and infrastructure but really wanted to um go the full stack

route so I thought marketing engineering would be really good build beautiful websites nice nice amazing and you're

when you went to UT right in Austin do I have that right I see some folks are in Austin I'm in Austin as well it's a

beautiful windy day here um so somewhere else where we have two folks located is Joe and Sky who are with us from stats

EG Joe I believe that you guys are all in Belleview Washington in an interesting piece of information that

I've never fact checked before but I just someone just told me this and it's it's fact in my head there are more

billionaires in Belleview than anywhere else in the world per capita so tell me if that's right or wrong and then also

tell me your Heroes journey I want to know how you've gotten to Stats Sig what your story is what your role is at Stats

Sig and kind of what you're up to yeah B you um I mean I personally don't know any billionaires would love to become

acquainted with some maybe get out more maybe get to Starbucks I'm sure they're hanging out there I'm sure they're there

yeah but yeah um yeah we're based out in Belle I'm actually from uh Toronto and uh I went to University of waterl um

I've been at statc for about three and a half years now and also started on infr team um here getting set getting like

all our services set up and optimizing all our different uh like backend payloads um and then now I transitioned

over to the Enterprise team where I work more closely with customers to build out feature requests for them and then also

um work on like partnership work amazing yeah and you've been at static for the whole kind of ride right like you were

there early yeah I started like before we had any customers and um really just saw like us build out the whole flow now

and here we are amazing okay and with you from stats IG is sky sky give us kind of the heroes Journey on on on

where you're and where you're at in your career how long you've been at Stats IG uh and you work on the marketing site so

you're kind of a stakeholder to a lot of our Engineers um who would be you know consuming stattic of sell I'm curious uh

you know kind of your story as we head into this conversation yeah so I've been on stattic for about two and a half

years um definitely non-traditional marketing background um I was never a marketer before satat uh sort of came

from data analytics uh uh and then um yeah it's been a it's been an awesome ride I mean st's grown a ton and it's

really cool to get to tell our story and also just help a bunch of developers bunch of PMS uh build better products

and improve the stuff they're working on um I've also a little bit of web dev on the side so it's the stattic very

Scrappy and so uh you know I also care a lot about our website performance and have gotten to use stats IG and notion a

lot to improve our our own website so amazing okay great this is well you've come to the right place guy we're going

to talk about web development today is the day um enter a market or leave a fullsack developer this is how with you

know this is how we roll um so for those of you who have never met me or heard from me before my name is Matt Jared I

work on the sales engineering team uh at forell and it's effectively our job to help you know customers or potential

customers evaluate for sale uh and make sure it's a great fit for for what you're up to and what you're working on

um if you have not ever heard of Rell I'll give a little bit of an overview of that here in a minute we'll talk about

the ins and outs of you know how we can best help teams uh and then we'll get into kind of a little bit of a deeper

discussion um okay one more kind of tiny bit of housekeeping if you're unfamiliar with what we're up to today here's where

we're going um like I said earlier at the very bottom of all of this we're going to have a Q&A so if you have

burning questions throughout what we get to drop those into the Q&A or into the chat and we we have some folks in uh who

are working with us to kind of keep an eye on that uh but what we're going to do is we're going to go over an overview

of notion stattic and forcell so if you maybe heard of one of these or two or all three uh and maybe need to refresh

her we're going to give an overview of all those things and then we're going to hear directly from Carolyn how she has

used all two or all all two of these tools all two uh how she's used statting and versel um at notion to help with the

marketing engineering team that she helps run uh and kind of the things that she's done on the experimentation around

performance and some of the things that she's done to improve the performance of notion.com uh and you're going to kind

of hear like the whole story around that and then what we're going to do is we're going to turn on our developer hats like

I said we're the guy's showing up as a marketer he's going to leave as a developer and if you're a marketer you

might leave as a developer too Joe is going to work us through walking through the flags SDK by versel and how it

integrates with static so if you're interested in what Carol ends up to and you kind of are compelled to start going

uh there's going to be some collateral afterwards kind of like that you can go do this on your own and Joe's gonna walk

us through how to do this as well and kind of give us a a primer on how to get set set up and how to get kind of into

this testing and performance culture um at your company and as I've been saying over over if you have questions drop

them into the chat drop them into the Q&A this is what we're here for um so yeah so let's get into it there's a

couple things that I want to go over first um first is what is versell um in the easiest possible way to think about

what versell is is that it's a platform that helps fast engineering teams build fast websites right uh and what I mean

by all of that is what forcell is going to do is it's going to provide a platform for you to take your

application so your web application we're going to build that in a way that it's very easy for you to understand how

it's being built it's very easy for you to customize that and it's very easy for you to go collaborate with others so

your team's going to be fast and then what we're going to do is we're going to throw this into infrastructure for you

um a lot of times when you look at cloud deployments and things like that a lot of times you have to bring your own

expertise and there's kind of a black box that you kind of maybe need to know how to learn or like how to work you

have to hire experts to help kind of manage those things we sort of flip that model where instead of saying bring your

own expertise we say bring your code bring your application and focus your expert on that and on the infrastructure

side we're going to handle that and we're going to deploy all these things for you doesn't mean it's going to be a

black box that you can't see it's going to be a thing that you just don't have to worry about uh we're going to give

you as many insights as humanly possible that you can see in things and this is where the fast teams comes from so your

sites are going to be very or excuse me this is where your fast sites are going to come from though we have fast teams

our teams are going to have lots and lots of information about what's going on with their application what things

are going on before it gets deployed and then once our sites gets get deployed they automatically going to be fast

because versell is going to bring the expertise on deploying those things and managing those things and scaling your

site and things like that uh deploying it around the globe and this is where we're going to sit on the on the fast or

excuse me fast sites I'm getting get confused um so this is where verel sits um it's an extremely easy way to build

and deploy web applications right so this is one half of if you're on the marketing team is you know I want my

site to get up and running but what happens if I want to test things Sky Joe could you give me like a rundown of what

stat does uh I'm now you know I want to take a step further beyond my deployment and I want to test things I want to do

feature Flags um Joe you said you've been there for the whole time Sky you've been there you're very deep on the

marketing side give us a rundown of what stats IG is outside of being in Belleview Washington give us give us

kind of a rundown yeah I can give the short version then um Joe can add some more

detail if he wants but um yeah so that's like we're building the modern platform for product development um there's a ton

of workflows around releasing features measuring their impact uh looking at longitudinal data on your product that

are traditionally handled with a bunch of disparate tools um we're letting you do that all in one place with a unified

platform for experimentation feature Flags analytics session replays user stores and I bunch of other stuff um and

we're really built for modern fast growing teams um if there's a growth stage software company out there uh

including for sale and notion uh and open Ai and anthropic and character AI you you sort of name the company they're

probably using stat to make better product decisions and also just move way faster um and then performance is also a

huge part of everything we do um I think historically a lot of these tools uh introduce you know flicker or introduce

lag um and we've been really rigorous as a very engineering focused team on making sure none of those are a blocker

to actually using this product and you know getting more data into every product decision you make um Joe

anything you'd add there um I think you KN hit it on hit the nail on the head for who we are um I

will just plug our customer support as well like I think one of the big like differentiators of between us and other

competitors might it's just our our support you know like we're always doing our best to answer slack questions and

threads like as quick as possible and working with um other teams to you know actually solve the the analytics

problems that they might be having um that are specific to them awesome this is so neat so this is

more of instead of me kind of guessing if I'm you know maybe coming from like I'm building this muscle out for the

first time as a marketer I'm going to know what's going on and I'm gonna have a partner there with me which is stat IG

and I'm going to be able to deploy all this with for sale uh and I'm gonna have an idea of what's going on it's what you

said earlier Sky which we'll talk about here in a second is there's not going to be a like um a flicker of where our

tests are or where our flags are executing we're going to do this in a very performant way uh so we're going to

talk about all this here in a second but before we do that I want to talk a little bit about what Notions up to uh

if someone maybe hasn't the ever used notion Caroline I'd love to kind of hear outside of these cool little

illustrations that you guys have in your brand um give us kind of the rundown on what notion is how it works you know

what type of team would use notion definitely so you can think of notion as an all all inone productivity

tool anyone can use it um even if you're just um single player planning your vacations to uh large companies like

some logos that we see on the screen um we did get our start um where you can just take notes but then quickly grew it

up so many other products um like wikis project management forms sites like even calendar um so I like to think of it as

a place where you can build the tools that you'd actually want to use and collaborate with others easily uh and

then yeah my my tagline is always is has always been if you can dream it you can do it on notion so love it yeah I mean I

love it I the notion calendar is one that we I've used for many years it's really really useful and so we use

notion uh in and out um you know for doing documentation internally and things like that so very familiar um but

one thing I want to talk about though and this is let's get into the meat of it here is you work on the marketing

engineering team um it may have a different name internally uh but traditionally you're GTM engineering

something like that traditionally what we're looking at is here is our application that's one piece of this and

then there's a whole arm and there's a whole network of other sites such as documentation such as blogs you know

main marketing P Pages solution pages tons and tons of different things maybe Community Pages things like that they

kind of span in the marketing engineering world and this is you know maybe where Sky works on that side is

there's just this huge network of pages and things like that that kind of are entangled in in what you have to kind of

work on and in particular um Caroline what you work on is the notion.com site and Is this different than like

notion.so and and what's the team structure kind of look like um you're working on just notion.com

right exactly uh and fun fact a year ago uh the marketing site and the app were both on notion.so but we've since moved

the marketing site to notion.com uh lots of reasons for that but uh the marketing site is Billy hosted on versel uh so

we've really enjoyed the collaboration here and yeah nice okay great so notion.com you guys split out to where

like the app is in this and notion.com like this is now our new kind of focus um and when you launch over to

notion.com this is a year ago you're kind of in seat you know we all have been there

when we sit at a new job we're trying to you know figure out up from down figure out where we sit what type of stuff was

kind of going on when you first started in terms of performance how was the site performing what did it look like um what

were type of what were the type of things that were happening with the notion.com site uh and in particular was

there anything where I was like hey it's time for us to really take performance seriously like what was what was that

like when you were first getting settled so uh yeah when I first started uh we kind

of had a good breath uh for on the marketing side we had a bunch of landing pages where people can discover us a

pretty good SEO and then we started to shift our gears to Quality so historically notion may not have been um

the leading uh company in terms of marketing site performance so we really wanted to buckle down and tackle it and

uh this was really uh an engineer L effort um we essentially introduced the fact that okay yeah we can start a

project where we move to notion.com we knew it would uh be better for our brand and also for performance and after we

launched that um we've steadily been tackling more performance uh projects so it's been uh slow but sure journey to

get us there nice awesome okay so let's get into it here so you you know we've got to fix things we've got to make make

it a little bit better maybe there's like our taste is at a level higher from what we're performing right now what was

it like when you were first sitting in like trying to figure out the first few things to work on so like you're using

um you know certain tools to figure out what you're looking at what was it like and what tool did you use that kind of

screamed like hey we got to fix this or we're gonna get you know our you know our paid traffic isn't going to convert

as well as it should uh our SEO is maybe going to be hurt because our core web vitals aren't performing as they should

um how did you how did you measure all of what was going on what tools did you use and were there any early wins that

said like oh I've got to fix this one thing uh we used uh kind of the industry standards so things like Lighthouse

score Google page speed insights uh to really get us benchmarked um around how um our speed is doing objectively and

from then on then we can formulate the plan on exactly what uh were the main bottlenecks nice and any any in

particular like lwh hanging fruit that you just like immediately knocked out and could see

improvements the biggest one uh was definitely thirdparty scripts so uh for a marketing website to run um there's

lots of uh third parties in terms of tracking analytics that we need to do and yeah marketers really want that

information uh but then yeah depending on how it's implemented um it could end up blocking the website um yeah and

sometimes things go forgotten so there's some unused ones that may be blocking the website for no reason so that was

definitely the biggest lows hanging fruit nice and what was that conversation like with marketing when

you were like hey I know we love this Tik Tock pixel but this thing's slowing us down we got to get this thing out of

here or I love this you know Pinterest pixel because it's giving us you know maybe a little bit more Rich data for

the paid ads we're running over there but we gotta maybe delay this or defer this what was that convers

like so engineering and the marketing or uh it really helps that we have the same gr goal um same Northstar metrics uh

which is conversions so yeah we can um go with the data and say like oh yeah if we improve um performance by this much

then it'll get us converting um better by this much and so that can really start the conversation to um put hours

into this project um real people's hours because yeah it's not really easy to align people from different Orcs so um

yeah awesome any any big kind of like pop of funnel changes that came through that were like hey we took third party

scripts off that really bumped us up and then like top of like you know people were really like Blown Away by how many

new leads we got or how much traffic was coming through um anything to share on that side like even if it wasn't third

party scripts were there any other kind of big like business wins that came from improving performance

performance uh we did uh we did see a noticeable difference um even in the order of seconds especially on mobile um

and uh connectivity like 3G so uh it was a really big one there and we were able to measure uh we were able to run all

these performance uh in terms of experiments so we were able to measure the top of um like the Topline metrics

that everyone cares about so super cool nice okay so there's one thing that you do differently than I

would say most places and where you guys have at notion is an ie for experimenting around performance um so

how do you do that in terms of you know I've been on marketing engineering projects um through my whole career

where I've gone from hey our site's broken it's really slow let's figure out what's going to happen let's just look

at like where we are you know the first step of every single performance project is to go get a bunch of data like you

have to know where you are to know where to go um so let's go get some data let's figure out what's going on and then we

just go like rif off a bunch of tickets or just like figure out what's wrong and just Hammer a bunch of code um and then

like hopefully the numbers would get better like number go up right like that's what we were hoping for um so and

we've had a lot I've had a lot of success with that over the years but you guys have a little bit more of a novel

approach where you're looking at this through let's test this with stats IG um and what stats IG is going to do is it's

going to be able to like give different experiences to our end users so let's say we're going to test uh having like a

slow loading like image on the top of the homepage uh this image is going to take like you know a huge amount of like

like space but it's going to be like for you know some sort of you know event that we're having and we want to have

folks come look at this event we're going to funnel this into maybe half of our users just to see if they're like

having a good experience or not having a good experience and then you're going to use staig to take those like that data

and that learning uh and then go and then ship further updates on that to to go have more performance improvements

around that um what is that kind of culture like on your side Caroline when you're saying like hey let's go do a

test around is this going to help our performance or hurt that performance what is the process like for getting

something out the door so we uh lean towards having a proof of concept before investing

engineering time and one of the things we did uh which may may not be a spicy take is we did um an experiment where uh

we intentionally so slowed down the website for a portion of our users um to really show that um yeah like every

millisecond matters every second matters um so kind of having that culture where we're not afraid to try things obviously

not at the expense of user uh user experience but um yeah being um being open to gaining data that we can uh

better use to inform future uh future positions nice so very data driven and this stattic is going to allow us to be

data driven allow us to sort of pilot some of these things uh see if it's going to work see if it's not going to

work uh number doesn't always go up right like sometimes our experiments and our hypotheses don't always you know pan

out the way we want to that's why it's important to be able to quickly shift to other things um this is you know one of

the very unique use cases for where static is um how often I guess Joe and Sky how often do you guys see teams

taking this approach um I know stattic you guys can do so many different things is this a common approach would you say

um for folks to use this to sort of understand their like how their sites performing or do you see this more in

like a where stattic users are more in like feature Flags like what's what's the common use case yeah I think that's

a very common use case um we see a lot of our customers add like performance metrics to every single one of their

flags and experiments um through our guard roil metrics or even through setting them up in our experiment

templates so that it kind of becomes like a culture amongst the organiz ation um this just allows them to see like for

every single flag or experiment is am I regressing performance and then being able to roll back any change that they

see that is actually affecting performance metrics and then we actually take this a step further as well by

providing meta analysis tools so users can actually look at for for a given performance metric what Gates and

experiments are actually affecting it so they can see like oh these Gates uh these experiments actually improve

performance so we should do more of these or oh these experiments actually uh have regress performance runs pretty

heavily so let's stop doing those and maybe roll them back that's so cool uh and just kind of

to get into it a little bit more you guys are like on the full stack too right so it's beyond like in this case

we're just talking about web experiments but if you're using feature Flags you know maybe on an API lay layer or

somewhere in a mobile app you can still use stattic there too right do I have that right yeah yeah we have teams that

um a great example is uh like open AI uses this for you know experiments in chat GPT all the way to like

infrastructure changes or controlling you know their API services and they're running experiments to optimize

performance on all of those Services all the time um so people use us literally from everything from you know

orchestrating API Services all the way to like running marketing website tests and uh it it's you know reliable and

quick uh for all those use cases awesome yeah I love that so and this is where it's like I'm you know I'm deeper into

like the tool chain of building this application right I have like I want to test the responses on this API I have

two different endpoints uh let me actually do some logic inside of stattic that's going to you know flag these two

you know use cases in uh or I have a mobile app or you know I'm chat PT and I want to change you know this user

interface things like that um to kind of piggy back off of where Caroline was and where the Notions using versell I think

I want to dig into just one really quick kind of point that versell looks how we look at for performance uh and if you

all at home or you're in an office or whatever uh this is I think one of the main kind of drivers that I would love

for you all to walk away with is what it looks like for the life cycle of a request uh in what Caroline's team is

actually doing behind the scenes here is leveraging the versel edge middleware uh and we're going to talk about how to

like actually run this yourself here in a minute when Joe when you know kind of gets Cod uh breaks out the code editor

for us here um but the idea here is let's just like walk through every single step that were to occur in this

situation and this uh I think is the to run flags with stat with saig and to also do it in a way that has no negative

performance uh implications right so this is going to help us on the performance side it's not going to hurt

us on the performance side so from top to bottom here let's look up top uh the client requests acme.com so notion.com

ver.com whatever website you want to go to a client's going to request that so you're G to like click on a social media

link or you're going to type it into the URL bar and hit enter when you hit enter that request goes off to a server or to

a CDN uh and there's now there's a life cycle of that request and we have a really great blog post on this that

we'll share out afterwards and that request is going to go to that origin server and then it's going to request

and there's going to be logic returned or it's going to hit a CDN uh which has you know maybe static assets and then

it's going to return that experience for someone right what verell does and what we do is that's really really special is

we have this Edge middleware and this is going to effectively intercept every single request without having any

latency impact right so we're going to request this mle middleware this middleware is deployed all across the

globe we're going to request this middleware uh we're going to talk to Stats IG and we'll talk and Joe's gonna

explain how how stattic and verell talk to each other here in a second uh and this is going to decide and staic and

versell together are going to decide which experience someone should go have so if I say I'm in group a then I'll go

over here on this dotted line over to this experience if I'm in and this is the control right so if I'm in group b

then I'm going to come over here on this little purple line and I'm going to now have this blue this purple button and

we'll talk about how to build this uh build this out here in just a second but this is the idea is what Carolyn's team

has done is they've tested the performance of this and to be able to do that without having client side tests

running and to be able to do this at the middleware layer above where the a normal request occurs gives us a ton of

power because we're still going to have very very fast web experiences we're going to write a little code that's

going to create that different experience and we're going to have those two basically identical web pages

outside of small changes with the tests um and really quick Joe I know you're going to get into this a second but like

could you talk to me about how like how stats IG views this middleware here because this is what we provided for

sale and then stats IG sits on top of this and this is where in we'll talk about this in stattic I can kind of

configure all of my tests but what does stattic do inside of this middleware layer can you kind of give us a little

bit of a breakdown yeah for this use case we're really leveraging the middleware to basically do our

evaluations in memory inside the middle World layer so what we're actually doing is we're taking our like flag

definitions and experiments and pushing them directly to the edge middleware um where the edge middleware can actually

read from the edge config that we're pushing data to um basically all in memory and then doing evaluations within

memory since our sdks um uh are able to evaluate um all all all within there um and then from therea we actually get the

flag evaluations that is just up to uh We've then like just use some logic from the new Flags SDK um to actually

redirect users to a cache version of the The Experience within their CDN um maximizing

performance amazing um okay so here's where and I'm seeing a little chatter in the chat I just want to kind of address

this is like the killer ability on all of this is for us to build this and we're going to build this experiment

here in just a second together but the idea behind all of this is that this is all done at the request so this is not

going to be done at the client side if we were to do like use a thirdparty service uh on our website that actually

comes in and changes stuff around for our for our tests right like you would think that that's actually better

because I'm going to be able to manage this in like a goey a graphical user interface I'll be able to kind of figure

it out and I'm gonna slap this on my page this actually really really really kills performance and there's two

reasons why first is the cumulative layout shift right so we don't really know what that test is going to do on

our site so if I instead of let's I was going to put my hand on my screen so like let's Imagine This this is gone and

I'm only going to jam on my like jam this button onto my page there's a cumulative layout shift that occurs

which is really really really really bad for users those users are looking at a page and then all of a sudden a bunch of

stuff changes around like I'm disoriented I'm confused there's already lot million things going on in my brain

right now I have no idea what to look at was this button just flew in out of nowhere what is that that's really

disorienting for users so we don't want that we want to test at the middle War layer the other side of this is that

it's really really bad for search and optimization because SEO crawlers and Bots and things like that are going to

look at this and they're going to say here's our site but then a button gets jammed in and it ships around there's a

bunch of layout shift is what that's called really really really impactful and really harmful for our SEO so for

our core web vitals so if notion were to do this without Edge middleware in stats EG they actually would have a really

hard time with their SEO because their SEO is run against really relevant content and the other side of that is

your search results actually run against your corad vitals so if your core web vitals are poor and they're damaged by

running client side tests then you're actually going to have a poor experience on the marketing side and your team is

going to actually have a PO experience uh trying to go convert leads or trying to go spend ads that are going to like

um you know convert into actual dollars that come back to your site so here's the idea is what we want to do is we

want to push these tests into this this middleware layer we're going to show how to do all in code andjust this second I

promise it's not too scary but what it does is we're going to lean under yourell and statti to build all this out

for us and it's a really really powerful pair so when it gets into the point of where we're actually and I'm actually

going to come out of out of speaker mode here uh and I'm actually going to pull open a site for us and when it comes to

managing all these things and when it comes to saying like well I have a bunch of different you know I have a million

different applications that I'm trying to run and a million different tests that are going on how do I keep this all

under control and how do I keep stat Sig and versel talking to each other in a performant way where I understand what's

going on and what we've done with versell just launched recently is this Flags SDK and what the flags SDK is

going to do is it's going to connect directly to Stats and in stat I'll say I have a bunch of different experiences

that I'm going to try and test out but before I get those up and into my site I need a place to test those right so

let's go back when I was talking let's I'll actually show you guys let's go back to when I was talking about versel

we really care about fast teams and we really care about fast sites those two things are really important so those

fast teams are part of that is I need information really really quickly so let's look at this side inside of the

flags SDK this is going to exist inside of our toolbar which I'll show you here in just a second and what it's going to

do and I just want to demonstrate this really quickly so we can see where it is is what I want to do is I have some code

behind the scenes that's going to change this dark background to a light background if I were to do this in a

traditional environment not using stats Sig and the flags SDK I would have to do a bunch of stuff in my URL I'd have to

go fight a bunch of cookies I would have to go to different branches I'd have to talk to other people have to set

meetings i' have to think about stuff i' have to slack people questions but what we're trying to do is eliminate all that

we're not trying to eliminate Communication in collaboration what we're trying to do is make that

collaboration and communication more fruitful so what I can say is I want to enable this dithered hero flag like nah

I want it to be white so what I'm going to do is flip this out and what's going on with the flags SDK is if we go back

to this where' I go if we go back to this life cycle of a request is I'm flipping between these two experiences

back and forth and it's showing me and this is now on a site that you all can experience but this is a little bit of a

preview of how it would look locally and Joe's going to show us how all this works here in a second same is true if I

want to change change some text around uh same is true if I want to enable a banner and all of this is going to be

available to me to test things before it gets to production and this is really helpful because if I'm a developer and

I'm doing you know 30 tickets a week and I'm cranking around flying on all these different things it's kind of hard to

keep it like keep track of what flags are in what place and how the ux of things is going to work much less if I'm

a stakeholder I need to know what my tests are running how they're going to work how my ux flows and things like

that are going to go together so where this all occurs is over here in this toolbar on the versel side so this

toolbar is going to be available for all preview environments uh and if you're interested in what preview environments

are things like that look like um hit us up afterwards and we'll be able to kind of give you an example of how that works

but on this side here's where I'm actually going to be able to go to the flags explore and inside of here I can

show all my Flags same thing here enable Banner flag I can turn that to true and this is now going to turn this to true

for me and now I'm going to be able to flip this on so next is I want to go to here and I can actually see a bunch of

different things so the piece that we're getting at and here's layout shifts I was talking before about cumulative

layout shift there's none here the flags SDK is doing its job it's preventing any layout shift from occurring I'm leaning

on the middleware that's actually pointing me into the two different experiences that I showed and Stat 6's

driving all this for us right so what I want to do is I want to pause from here and I want to get like our hand dirdy

and I want to show everybody who's with us now out how we can go do this on our own and we have we know we have you know

20 15 minutes left and we're going to be able to go build something out so I'm going to take the screen share off Joe

could you uh could you take us away on like if I want to go build this from scratch what do I do where do I start

let us know what page you're at right now looks like you're on the marketplace for versel yeah yeah so we're gonna get

started by actually just heading over to vel's Marketplace um and you'll actually be able to find stat zig's new

integration right here um so if you scroll all the way down on our listing you can actually

follow along in this demo by using the new pre-built template that we have here um you'll just be put into a little flow

here where you actually create a brand new stats uh a brand new for sell project and then you'll be prompted to

actually install the integration we've really optimized this flow to hope to have startup be as quick as possible so

we actually here will enable Edge config syncing and then also create a stattic account for you as part of the

installation flow and then also link the two um projects together by setting and end variables inside of the project that

you're deploying so all of that should be handled for you and then you'll also so you'll be able to use the sdks right

out the right out of the box cool after all that set up all you got to do is hit the pl and you'll

basically have a running website using ver cell in static I'll just head over to uh my

local copy of it um so that we can look through the example here um we just have two sample Flags right here which are

the summer sale banner and then this free delivery Banner um and then you can see here as we oh there's a button on

the bottom right that allows you to um basically reset your user so that you can get put into different experiences

and as you can see here um the different banners render in um immediately with no layout

shift um we also uh integrate with the flags explorer that Matt just showed so you can see here um there's a button

that will actually deeplink you in to the stat Sig and you can actually see the setup of the gate so you can change

change things really quickly right now we just have a really simple 50/50 split so that any user that lands on the page

will either will have a 50-50 chance of actually seeing the banners um but you can edit this really quickly and then

also look at uh live insights oops wrong look at live insights of uh the checks that are coming in um you can see some

of these checks coming in live cool so that's the demo project that you'll have up and running um so

I'll actually just run through some of the the code now um the code uses Flags SDK backed by

the stattic adapter that's built for it um and really to get started it's a pretty simple process of just

implementing two functions uh the first one being the identify function from the flags SDK so

what this is looking for is what user you're actually evaluating flags for um so in this case we have a very simple

identify function that just grabs a stable ID from cookies that are set on the request and then crafts a user ID uh

crafts craft user object um but of course if this is also where you would add other properties like what country

the user's from or other IDs that are relevant to your business and then all you got to do is

actually Define the flags so here you can see we're setting two flags the summer sale banner and the free delivery

Banner having the saig adapter back that and then uh passing the user object that actually will identify the user and then

from here that's all you really need to do to get started with using the flags you can call these functions and then

actually use the flag evaluations within your project to actually um render different

experiences um but as we said we need to go further and actually show the pre-compute pattern and how we actually

Leverage The middleware to actually show the static experiences so I'll head over to the middleware now and show the brand

new pre-compute pattern that was implemented in the flags SDK um So within the middleware we're

actually using this function called precompute which takes in the list of flags that you have defined and actually

will generate a code that will be used to redirect users different experiences um this code is going to be unique for

whatever evaluations that the user gets um for their flags and then this is how uh we actually can direct users to their

unique experience based on the flag evaluations so we'll rewrite the user to a to their own unique page and then on

the actual um component um we'll use the get precomputed function pass in the code and then actually get the flag

evaluation so you can see here given the code we'll now get the summer sale banner and the

free delivery banner and then use the results to actually render to decide whether or not to render the banners

inside of the application um this should be just a onetime setup and you should be able to

just um add your middleware uh add these functions to your middleware and then to your uh components and then have your

flags be able to be used throughout so um should be a pretty simple one time setup um I will put is crazy oh yeah

yeah for sure um this is so cool go ahead sorry what were you gonna say oh yeah I was gonna point out one other

thing which you might notice that um there are going to be a lot of different variations of pages and you might think

oh um this might like cause like uh you know a lot of cach Misses because users might um be redirected to different

experiences that other users have not seen before so you you're going to miss the cach um but this is actually where

for cells incremental static regeneration pattern comes into play so so if you look at our layout. TSX file

in the example there's this function called generate static prams and this is actually where we'll actually pre-build

all of the different permutations of the ex flags and experiments that you're running and uh you can see here we have

this generate permutations function that'll actually uh basically allow us to know allow the build time to know um

what different variations of the page that it needs to already have ready to go and then when users come um they'll

actually be hitting a pre-built page but in the cases where you didn't pre-build it um it'll actually build it at runtime

and then uh subsequent request will actually be served from the cache and the CDN still um again maximizing

performance there this is so cool um I really want to double click into two things the

first one is the initial setup work this is so if you're using ISR for generic setup Brands this is one thing if you're

not using that just yet um I think it's just in the page and just in the middleware or no excuse me it's just in

flags and just in middleware and you set this up only once do I have that right um so you only have to do this once for

when you're like connecting to stattic in your code base um this is where the flags get set up and then in the

middleware which we talked about before this is basically just saying like hey go off and get my flags and my flags are

G to get cashed uh and you're gonna get deployed on the globe with for sale and then this is all just set up once right

yeah exactly this is this is a one time set up like um we we've like in the example you can see here that we're

actually passing in like a set of flags um so you'll need to only set this up for your initial set of flags and then

uh and subsequent Flags like you'll need to of course you'll need to Define them and then use them but then once you add

it to your product Flags you don't need to make any more changes to Middle word because that pre-compute pattern takes

care of all the different new variations that come up okay amazing so and then my next

question is if I'm changing these or I'm doing new tests or you know this one's like proven that it's going to work and

I want to implement this all I have to do is essentially say like hey I want to do a you showfree delivery Banner flag

like show new promotion flag create this and then I pass that value down and I think if you go two tabs to the right

like what you're doing here is you have this component so stat said gate exposure or free delivery banner and you

basically are passing a prop that says show or don't show and whether in the the solution is show free delivery

Banner so whether that show free delivery Banner is true or false then we're actually going to show or not show

this flag and that all that logic on if it's going to be 50% 90% 2% whatever it is this actually all gets done at the

stats EG level and then it just kind of trickles down into our code and then once it's in our code code then it's

built and deployed and off to forell then it's just going to evaluate that way and then after all that works so let

me pause does that do I have that right Joe that kind of flow of how this works from stat toell and then and then into

our end users do I have that right yeah that's exactly right and all that logic that you just described is handled in

this pre-compute function here um where we're actually leveraging vel's Edge config um like we're pushing our configs

over to vel's Edge config and actually doing all the evaluation of the flags that you just mentioned um right in this

function so the user doesn't have to think about any of that stuff amazing and then if I want to test this or say

like hey product manager hey designer let's play around with this then the flags SDK automatically picks each of

these up and then it allows me to like toggle on and off do I have that right too yeah exactly we'll push um any

updates that are made to flags and experiments right into forell um without any user interaction needed right

amazing and then you have all the best in for sell with like in terms of preview comments you can do everything

for sell and then all the analytics like the deeper analytics that are provided by stats IG that's all straight out of

stats IG um is how that would work yeah exactly amazing as I saw as I showed like the integration is super

easy to use you can just literally go into your gate and look at uh the setup and Diagnostics and also analytics um

right from the toolbar so cool and let me just kind of look back to Caroline really quickly was

this Carolyn is this kind of how your team operates uh I know that you guys are heavy users of the flags you know

the flags SDK and the toolbar at notion this is kind of how your team works right yeah very similar yeah so you're

going through evaluating code seeing what's going on getting information and then making like using the that

information to make decisions and move forward so this is so cool um okay questions Q&A I will flip us back really

quick before we get into Q&A I'll take back over on share um before everybody hops off if you have questions go ahead

and drop those into the questions uh the Q&A box and Carolyn how can we keep in touch with you um if anybody's gonna

bounce early how can we keep in touch with you LinkedIn Twitter anywhere in particular that we should be following

you uh or following notion notion.com go to notion.com a bunch and see what's an experiment or what's not an

experiment I would say all the Notions social media is way more interesting but I am on LinkedIn um but yeah follow

awesome everywhere like X Tik Tok amazing amazing okay cool and sky and Joe really quick I want to make sure we

get a plug for you guys how can we follow along with you individually what's the best way to kind of keep in

touch with what stats up to and what you two both up to as well LinkedIn and Twitter I think uh for

both both of us and for stat awesome and it's at stat on okay okay great awesome and

we'll follow up with all this um yeah so questions let's go to the first one um I'm gonna answer this one live how does

verell differ from render and can you deploy versell apps uh in our own infra AWS so the first question for this one

versell differs from render in in one key area and what versell specializes in is deploying nextjs apps and deploying

front end apps uh you may know us as a front-end cloud or a developer Cloud where we really focus is full stack

applications in the nextjs and frontend ecosystem uh is where we started but now we've moved on to larger kind of like

more apid driven fullon SAS applications stuff like that uh where render really shines I think is uh with other

Frameworks I don't really know that much about render but what forelle is going to really shine on is building nextjs

applications from scratch without you having to bring any expertise um render does other things um in the other side

of that is can you deploy versell apps in your own infra answer to that is no versell is its own infra and nextjs is

what you would take and go pull that and then you would go run all of that by yourself on AWS uh again so the idea is

with verell it's bring your code and we bring the uh infrastructure and we bring the expertise with other platforms

you're going to bring your own expertise uh and then they're just going to provide the INF so that there's kind of

a flip in that equation is where it goes um and then let's see are you guys planning on using the newly

introduced no JS runtime middleware if yes on which scenarios can this be beneficial in comparison to the current

implementation Joe guy you guys uh I want to address this one uh on I would be interested in more for your guys take

on this um because no JS runtime middleware is something that's being worked on we can talk about more of that

offline uh and we can we can get in touch with you but Joe Sky do you guys have any hot takes on on nodejs

middleware for stats IG um I do know that we have two separate SD dks um one that's built

specifically for the edge runtime since there are some missing node um uh nodejs functionality Inside Edge run time um so

it would allow us to run our full sdks and potentially provide more feature set to the edge um so that would be great um

so yeah definitely approve of that more features yeah more fully featured when when node middleware comes out uh

amazing okay cool um next one is it possible to set us up with a self-hosted nextjs setup so I answer yes it is

possible uh but no you don't get the same experience so let's go back to this page right here uh you would essentially

have to go figure out the middleware on your own right like the middleware layer you could kind of go you know do some

routing inside of AWS there's a number of different products that would maybe be able to help with that I don't know

what they are um but this comes right out of the box and uh what Joe just showed us that whole kind of like

integration layer uh is a stattic integration so I don't know Joe Sky you guys integrate with ad us um outside of

this um not going to say it's impossible but for it would be again roll your own you have to figure it out on your own

we've solved that problem you would have to go solve it on your own but Joe Sky you guys have AWS you integrate with

other folks too right um we don't have uh like a native integration with AWS but like like you

said like um we would rely on customers to kind of build it out like the syncing mechanism of configs into verel I guess

aws's version of edge config maybe something something like a reddis um themselves yeah yeah so you would have

to do all this on your own um the nextjs part of it like where the code sits and how that code generates pages and how it

renders data and how it consumes content things like that um that would be something that you would have like that

you would get out of the box with next all of the other deployment stuff like that you'd have to figure out on your

own um so I'll go ah a and answer that um Joe you might know the answer to this um does the ID or it looks like

somebody's actually answering this one uh what's the biggest differentiator from post hog here should I switch um CU

Sky I wna I want to hear some answers on this one curious yeah I can take that um so pretty similar I mean I think our

vision is pretty much the same of like every product team and engineering team should have this like complete set of

tools that are all linked together and work really well together um I think the biggest difference is like scale and

scalability both from a pricing and from like a infrastructure standpoint so um we've built stats Sig to handle like

chat gbt level scale so hundreds of millions of users a month um and our pricing scales with that um so if you

compare like how much does it cost to use post hog as you scale versus how much does it cost to use stat Sig as you

scale stat Sig's way more affordable um and specifically for feature Flags like post hog charges you for every gate

check um and our gate checks are completely free unless you're using feature flags as an AB test so um I

think that's the biggest difference and then also on like the experimentation side we've gone like very deep to make

it very sophisticated we have a bunch of data scientists who poke holes and everything we do um so I think that's

also a big differentiator but um we love post hog they're an awesome product so no no shade there uh and I think we're

all building the same thing for engineers nice a lot of conversations about the beijan method and St have you

I guess Sky Joe kind of off topic any mathematical theories that you can share with us really quickly that you've

learned while working at static anything cra um like just beijan method I would

assume something like that I don't know that's the only thing I know about math so that's why I keep saying it anything

crazy that you've learned math wise that stat Zig has like blown your mind with uh I think the biggest thing is

like there's all the you know like deep deep statistical methods that you can and should use um but I think the the

reality is like there's always a level of judgment and like uh you know it's an art and a science it's not just a

science and so um I think you know you always want be able to see all the math and like dig in so you can actually you

know figure out for yourself what the answer is we we try to make it as easy and possible and use like the most

advanced task methods available but um you know every team's going to have questions and want to dig deeper so

that's always been our philosophy is like make it so you can explore yourself amazing so cool um okay there's

one in here I don't know the answer to I'm going to answer this live I know the Apple I know Apple limits user tracking

with stable IDs on mobile M so how do you circumvent that to get stable identifier for anonymous users this is

going to be out of the versell realm because that's going to be more Apple mobile stuff we're going to be looking

more just browsers I would assume is where this question is going but I wanted to read this one Joe Sky did you

do you guys have anything about Apple user tracking uh in stable IDs maybe we can get this one

offline yeah I don't have a like a concrete answer to that question but I know that R sdks do do their do the best

job they can to generate stable IDs for for you so like I'm hoping that like this is actually just built into our iOS

sdks where um it'll do it uh do its absolute best to try to generate one when

again awesome okay great um I think I'm gonna answer another one here just a couple other other questions kind of

trickling it I will watch this does it talk about how to add backend database like notion to your website built on

vzero thanks again um I think is kind of like a vzero question um I don't believe we have a notion vzero integration if

you're not familiar with what vzo is um I'll just take a minute and and we can kind of like do a quick tour uh vzo is a

way to use a prompt to build a full stack application uh so like I like make a web page that says thanks to

Joe Joe sky and Carolyn Carolyn am I spelling I'm spelling this wrong in three different

ways for joining us on a webinar today you what VZ is going to do is it's going to build me a nextjs application um it's

going to use Tailwind under the hood it's going to use Tailwind uh a styling Library called Shad Cen on top of

tailwind and it's going to write out an application for me um this is a fantastic way to get started and what I

would do is use this to get started play around with this and what you would do is be able to take this out of your

application drop it into versell and then again we've been talking about this keep leaning on this life cycle of

request and go ahead and take your site and say like I want a purple button VZ build you a purple button drop that into

your application and then test that with stattic on here uh and here you can see vzero let's see what it comes up with

thank you a special thanks to Joe sky and Carolyn for joining us on today's webinar I don't know it seems pretty

good I'll go ahead and deploy this to for S all right that's all I have for you guys I think that we're at the end

um Carolyn Joe Sky stats IGN notion you guys are awesome thank you so much for your time thanks everybody uh there will

be a recording of this uh shared out thanks everybody for your time keep in touch and uh let's ship some cool stuff

thanks everybody